Handwritten input in a restricted area

[0002] The present invention relates to digital electronic systems that receive and use handwritten input in the nature of text.

[0003] Handwritten computer input has traditionally been captured using a stylus and a graphics tablet. Graphics tablets have a flat surface on which a user makes marks with the stylus that become electronic signals representing the successive positions of the stylus on the active area of the tablet. Many tablets, such as the graphics tablet products available from Wacom Technology Corporation of Vancouver, Wash., are pressure sensitive and provide pressure data along with position data. The Wacom tablets include tablets ranging in size from 4×5 to 18×25 (inches) and provide 256 levels (that is, eight bits) of pressure sensitivity. Other vendors provide larger and smaller tablets as well as tablets of different pressure sensitivity. Such tablets are generally connected to a personal computer to provide a superset of the functionality available with a mouse. In this configuration, position and pressure data are transmitted to an operating system or application program that ultimately generates display data on a display screen in response to gestures the user makes with the stylus on the tablet.

[0004] An input tablet can also be integrated with a display device, such as a liquid crystal display device. Such display tablets are well known from their use in personal digital assistant (PDA) devices such as the PalmPilot™ and Palm III™ organizers available from 3Com Corporation of Santa Clara, Calif. The input tablets on these devices are characteristically small, having an active area with a width on the order of two to two and one-half inches in the direction in which the user generally writes.

[0005] An input tablet can also be integrated with a keyboard or notebook computer, or added to a computer as a separate device, as a substitute for a mouse or other pointing device. In some products, such input tablets are also pressure sensitive.

[0006] With some input devices and software, marks made on a tablet can be erased or edited after they are made. The Wacom Erasing DuoSwitch® UltraPen® stylus, for example, has an eraser end that can be recognized as an eraser by device driver software and application software. Input tablets that are used as substitutes for a mouse generally support click and double-click functions. PDA devices generally support editing functions that can be invoked by gestures made by the user on the device's display tablet, either in the form of editing marks drawn on the tablet or in the form of gestures that select editing buttons.

[0007] Handwritten input in the nature of text is generally written in lines of script or characters, either horizontally or vertically, and either left-to-right or right-to-left. The length of a handwritten input can easily exceed the writing width available on an input tablet. The invention provides methods and apparatus that allow a person using an input tablet—in particular, a small tablet of the kind often found on a PDA device or a notebook computer—to write and edit arbitrarily long text in a restricted area.

[0008] In general, in one aspect, the invention features methods and apparatus implementing a technique for processing handwritten marks made on a machine-readable tablet. The technique includes providing multiple regions for input on the tablet; receiving from a user handwritten marks made in a current region of the plurality of regions; receiving from the user an indication that the current region is complete and thereupon selecting a different one of the plurality of regions as the current region in which to receive handwritten marks made by the user, whereby marks are received in a sequence of at least two regions; and representing the handwritten input as the aggregate of the marks written in the sequence of regions in a writing direction, the marks in different input regions having positions with respect to each other defined by a logical concatenation of the regions on the sequence of regions. The input can be arbitrarily-long markings of any kind.

[0009] Advantageous implementations of the invention include one or more of the following features. A bounding box is defined for each input region and the bounding boxes are logically concatenated to define the positions of the marks in different input regions with respect to each other. Each bounding box has a length in the writing direction, which length is defined dynamically to fit the marks made in the region. Each bounding box has a width perpendicular to the writing direction, which width is the same for all bounding boxes. The bounding box for each input region has a leading boundary the position of which is fixed with respect to the input region and a trailing boundary the position of which varies with respect to the input region to accommodate the marks made in the input region. The input regions and the sequence of input regions have a writing direction but no other predefined structure with respect to the marks to be made within the input regions. The logical concatenation of regions includes logically concatenating a trailing edge of one region to a leading edge of an immediately subsequent region. The writing direction is horizontal or vertical.

[0010] Handwritten marks can be received at any place and in any arbitrary order within the current region. Handwritten marks are displayed substantially simultaneously with the making of the marks. Handwritten marks are stored as mark data in a memory operatively coupled to the tablet, and the stored mark data is combined to form the representation of the handwritten input. The mark data represents the marks made by the user as set of times and positions of a stylus on the tablet. Both a recently-written portion of the handwritten marks as made by the user and the marks made by the user in the current region are displayed while marks are being made in the current region. The tablet is a display tablet having an integrated display device. The recently-written portion of the handwritten marks as made by the user is displayed at a reduced size in a reviewing area on the display device. Unmarked space is logically trimmed from one end but not from the opposite end of each of the regions before logically concatenating the one end of the region with a sequentially neighboring region. The plurality of regions consists of two regions, and the user indicates that the current one of the two regions is complete by making a mark in the other one of the two regions. The display of the current region is cleared after the user indicates that the current region is complete. Handwritten marks can include editing marks such as marks denoting a modification of marks previously made in the current region. The modification can be erasure. The handwritten input can be a signature. The tablet has an active area with a length of less than about two and one-half inches in a writing direction. The marks written in each region in the sequence of regions are combined in an output stream as a chunk of data. This output stream includes a sequence of chunks, and the locations of bounding box boundaries are specified in the output stream. The handwritten input is displayed in lines that break at input region boundaries. The marks are received in a first region and a second region alternately and concatenated logically to an output stream. The mark data in the output stream are displayed by reflowing the mark data in units of chunks in a display area. The display area is a rectangular area defined within an electronic document. The sequence of chunks can be edited by deleting a chunk, by inserting a chunk, by replacing a chunk, by modifying a chunk, or by rearranging chunks.

[0011] Advantages that may be seen in implementations of the invention include one or more of the following. The invention allows a person to write and edit text comfortably and reliably, even when the input area is restricted. The invention allows a person to enter a signature comfortably in a restricted area. The use of input regions imposes no constraints on what can be written or on the interpretation of what is written.

[0012] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features and advantages of the invention will become apparent from the description, the drawings, and the claims.

[0013] FIG. 1 is a diagram of a personal digital assistant showing a user interface in accordance with the invention.

[0014] FIG. 2 is a flowchart of a method for receiving and processing handwritten markings in accordance with the invention.

[0015] FIG. 1 is a diagram of a personal digital assistant (PDA) 100 showing a user interface in accordance with the invention. The PDA is a small, self-contained computer that has a microprocessor, memory, input and output devices, and interfaces for add-on devices. The PDA 100 runs an operating system that provides functionality and services for applications, that is, for computer programs configured to run on the PDA.

[0016] As will be described in reference to both FIG. 1 and FIG. 2, the display tablet of the PDA is controlled by software (computer program instructions) executed by the microprocessor and other elements of the PDA. The software implements a process 200 (FIG. 2) that prepares the user interface shown on the display tablet for handwritten input (step 202) by defining input regions 104 and 106 in the input or active area of the display tablet (step 204). Two regions are sufficient, although more can be used, and the regions need only be high enough (for horizontal writing) or wide enough (for vertical writing) to allow a user to write one line height (or width) comfortably. Optionally, the process can define a reviewing area 102 for the user interface (step 206). The reviewing area 102 is used to display, in a reduced size, what the user has most recently written. Displaying the recent writing at one-fourth scale in the reviewing area provides good visual feedback to the user.

[0017] The process 200 can be invoked by, or implemented in, any program running on the PDA that requires or can accept handwritten input. After having prepared the user interface, the process receives input (step 210) in the form of marks made by the user on the tablet in one of the regions (step 212). The process displays the marks on the display tablet where the marks were made (step 214) as the marks are made; or, if the input tablet is not a display tablet, the process can display the marks on a monitor or other display device connected to the computer running the process. If a reviewing area has been defined, the marks made in the current region are also displayed in the reviewing area.

[0018] Having begun writing in one of the input regions, the user can continue writing in the region by making marks at any place and in any order within the region. Thus, the user can go back and dot i's and cross t's. If the underlying operating system or application software supports erasure or other editing of marks on the input tablet, the user can use all supported forms of erasure and editing. The user can start over in the region by clicking the “clear” button 108 (FIG. 1), which causes the process to erase the region and allow the user to begin writing again. When the user is done with the current input region, the user can so indicate by making a mark in a different one of the input regions (step 218). The process then optionally trims the completed region (step 220) by logically trimming unmarked space from one end but not from the opposite end of the region before adding the marks in the region to an output stream. By trimming the unmarked space from the right of the region but not the left, the process allows the user to insert space intentionally within the writing by leaving blank space at the left of a region, while not compelling the user to write all the way to the right margin to avoid inserting unintended space. This provides a natural writing interface for a user writing left-to-right, and the same technique can be used, mutatis mutandis, for writing that runs in any other direction.

[0019] The trimming can be accomplished by defining a bounding box for each input region. To define the positions of the handwritten marks from different input regions with respect to each other, the bounding boxes are logically concatenated. The length of each bounding box in the writing direction is defined dynamically to fit the marks made in the corresponding input region. The width of each bounding box perpendicular to the writing direction is advantageously set to be the same for all bounding boxes and the same as the width of the input region. To define the length dynamically, the bounding box for each input region has a leading boundary the position of which is fixed with respect to the input region and a trailing boundary the position of which varies with respect to the input region to accommodate the marks made in the input region. As each input region is used or reused, a new bounding box is defined for the marks newly made in the region.

[0020] The trimmed region defines a chunk of data, which is added to the output stream (step 224). The display of marks in the reviewing region also reflects the trimming of the region (step 228). In this way, the region just completed is logically concatenated with the sequentially preceding region, and all neighboring regions are logically concatenated to form a representation of the user's handwritten input.

[0021] If the user has more marks to make, the process continues (the “no” branch from decision step 230) by repeating the actions of receiving marks (now in the newly selected region), displaying the marks in the region, and displaying the marks in the reviewing area (collectively, step 210).

[0022] In FIG. 1, the input regions 104 and 106 and the reviewing area 102 are shown as displaying a signature that was begun in input region 106 and completed in input region 104. In actual operation, the process would not display the handwritten marks in both input regions as shown. The process clears the display of the current region after the user indicates that the current region is complete.

[0023] The user can indicate that both the current region and the entire input sequence are complete by selecting the “done” button 110 (FIG. 1) on the user interface. When this occurs, the last region can be, but need not be, trimmed (step 220), as has been described. It is then added to the output stream. The process then provides the entire output stream to a program—typically, an application that requested the input (step 250 and “yes” branch of decision step 230).

[0024] The markings made by the user are stored as data in a random access memory of the PDA under control of the application and operating software of the device. The data typically takes the form of sample times and corresponding positions and pressures, if the device is pressure sensitive.

[0025] The output stream is maintained in memory in the PDA. The output stream is stored using a data structure, such as a linked list, that maintains the separate identity of the chunks, thereby preserving region boundary information and allowing programs that receive and process the output stream to process the output stream in chunk units. Alternatively, the output stream can be stored as a linear array or list of sample points with additional data specifying the location of region (that is, chunk) boundaries.

[0026] The output stream can be processed by an application program running on the device that received the handwriting or by an application program on a different computer that receives the output stream data. For example, an output stream in a Palm III organizer can be transmitted to a personal computer (step 250), such as a computer running the Microsoft® Windows® 95 operating system or the Apple Computer Macintosh® operating system, using the Palm Computing® Palm OS HotSync® architecture.

[0027] An application program can manipulate the output stream data in a number of useful ways (step 260). The application can fit cubic curves, such as Bézier curves, and splines to the data (step 270). The application can express the data in a page description language, such as the PostScript® language, by fitting the sample data points to drawing elements supported by the language, and the page description language representation can be rendered for display or printing (step 272). The application can display the data in any of the foregoing forms in a target region, such as a rectangular region within an electronic document, flowing the data into the target region in chunk units, which will cause the displayed representation of the handwriting to have line breaks only at the original input-region boundaries (step 264). Pressure data, if available, can be used to vary the thickness of displayed lines and curves.

[0028] The application can edit the sequence of chunks in the output stream by deleting a chunk, by inserting a chunk, by replacing a chunk, and by rearranging chunks (step 268). Either as a set of points or in a vector representation, each individual chunk can be edited using bitmap or vector oriented editing tools.

[0029] The application can also apply a handwriting recognition process to the output stream data to convert the markings data into text data (step 274).

[0030] The process 200 can be implemented using commonly-available software development tools for the platform or platforms on which the computer programs implementing the process are to run.

[0031] The invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Apparatus of the invention can be implemented in a computer program product tangibly embodied in a machine-readable storage device for execution by a programmable processor. Method steps of the invention can be performed by a programmable processor executing a program of instructions to perform functions of the invention by operating on input data and generating output. The invention can advantageously be implemented in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. Each computer program can be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired. The language can be a compiled or interpreted language. Suitable processors include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory and/or a random access memory. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks. Any of the foregoing can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

[0032] A suitable programmable processing system for implementing apparatus or performing methods of the invention includes a processor, a random access memory (RAM), a program memory (for example, a writable read-only memory (ROM) such as a flash ROM), an input tablet controller, a display device controller, and an input/output (I/O) controller coupled by a processor (CPU) bus. The system can be preprogrammed in ROM or it can be programmed (and reprogrammed) by loading a program from another source.

[0033] The invention has been described in terms of particular embodiments. Other embodiments are within the scope of the following claims. For example, the steps of the invention can be performed in a different order and still achieve desirable results. The input regions and reviewing area have been illustrated as running horizontally. For input in languages written vertically, the input regions and reviewing area would run from top to bottom on the input tablet. The tablet can be an input-only device and the marks made by the user can be displayed on a different device. The invention can be implemented on a palm, notebook, desktop, or other form of a computer. The sampling of the input tablet can be done at a variable rate as well as a fixed rate. A completed region can be trimmed to a bounding box around the markings, or the two ends of the bounding box perpendicular to the writing direction can be trimmed while leaving the sides of the bounding box parallel to the writing direction fixed. A trailing edge strip of the current region can be reproduced at the leading edge margin of what will be the next region so that the user can align and join markings across region boundaries easily and accurately. If the user does join marks across region boundaries, the pair of regions can be treated as a single chunk when reflowing the output stream. The bounding box, or markers identifying the locations of one or more sides of the bounding box, and in particular a marker identifying the location of the trailing edge side of the bounding box, can be displayed with the marks made by the user. The reviewing area can be implemented with scrolling, so that the user can scroll to and select a chunk of data, display it, edit it, and restore it to the output stream.