Resolving document object collisions

[0001] This patent application is also related to the following non-provisional patent applications: U.S. patent application Ser. No. 10/186,388, entitled “Writing Guide for a Free-Form Document Editor”; U.S. patent application Ser. No. 10/186,847, entitled “Method and System for Editing Electronic Ink”; U.S. patent application Ser. No. 10/186,874, entitled “Method and System for Selecting Objects on a Display Device”; U.S. patent application Ser. No. 10/186,837, entitled “Space Management for Electronic Documents”; U.S. patent application Ser. No. 10/186,865, entitled “System and Method for Automatically Recognizing Electronic Handwriting in an Electronic Document and Converting to Text”; U.S. patent application Ser. No. 10/186,820, entitled “Method and System for Categorizing Data Objects with Designation Tools”; U.S. patent application Ser. No. 10/186,463, entitled “Method and System for Displaying and Linking Ink Objects with Recognized Text and Objects”. These applications and the application herein are all being filed on the same date, Jun. 28, 2002, and are assigned to the Microsoft Corporation. The subject matter of each of these applications is hereby frilly incorporated by reference.

[0002] The present invention relates to editing electronic documents and more specifically relates to a method for resolving document object collisions resulting from document editing.

[0003] Electronic document editors are widely used in homes and businesses today. Familiar examples of these editors include word processing applications that operate on personal computers (PCs) and note-taking applications that operate on personal data assistants (PDAs). These applications strive to replace paper as the simplest means to record and communicate information. However, to replace and enhance paper's utility, the electronic document editor must allow the user to place document objects on a page and to edit, move, resize, and add objects, while ensuring that the user can clearly view the individual objects.

[0004] Note-taking using a free-form document editor can be enhanced by using certain hardware devices. For example, an electronic tablet can be used to record handwriting and input the handwriting to a conventional computer. Such electronic tablets typically comprise a screen and a handheld device that is similar to a pen (also referred to as a stylus). A user can use the pen to write on the electronic tablet in a manner similar to using traditional pen and paper. The electronic tablet can “read” the strokes of the user's handwriting with the handheld device and render the handwriting in electronic form on the tablet's screen or the computer's display as “electronic ink.” Additionally, the user can create a drawing or text onto blank areas on the page using the stylus and electronic tablet or using a keyboard, pointing device, or other input device. These areas of text or images are called document objects.

[0005] Some document constraints can reduce a user's ability to take notes efficiently and effectively using a conventional document editor. A free-form document editor operates to eliminate some document constraints by enabling free-form note taking. Accordingly, the free-form document editor has a different document behavior than a word processor document editor.

[0006] By eliminating some document constraints, a free-form document editor may trigger a need for object management in free-form document editors. By adding new objects to a page, or editing, moving, or resizing existing objects, one object may collide with another object on the screen. This collision could either cause one object to cover another, obscuring information in that covered object, or, as traditional electronic document editors do, merely bump the lower object vertically down the page (a one-dimensional solution). Neither of these approaches would allow the free-form document editor to maximize its two-dimensional character. Additionally, in some cases, a user may want the objects to overlap. As such, the system uses a method that resolves collisions two-dimensionally while allowing user flexibility.

[0007] Collisions between document objects on a document page are resolved. A collision is identified and a two-dimension resolution of the collision is provided by moving the object the shortest distance from a pre-collision location that would resolve the collision. The method calculates the shortest distance to move the object. The invention also may establish logic that designates some objects as “not able to collide” and the method will not move those objects to resolve a collision.

[0008] In one aspect of an exemplary embodiment of the invention, an existing page in an electronic document is modified. By adding a new document object or editing, resizing, or moving an existing object, the boundary of one document object intersects with the boundary of another document object. When the collision occurs, the method determines the shortest distance required to move the object to resolve the collision, either straight down or straight to the right or left, and moves the object in the direction of the shortest distance. If, by moving the second object, another collision occurs, the program resolves the new collision in the same manner.

[0009] In a second aspect of an exemplary embodiment of the invention, an existing page in an electronic document is modified. By adding a new object or editing, resizing, or moving an existing object, the boundary of one object intersects with the boundary of another object that the user does not want to move. When the collision occurs, the program does not move the second object. Instead, the objects overlap until the user reconfigures the objects such that the object is then able to move.

[0010] The aspects of the present invention may be more clearly understood and appreciated from a review of the following detailed description of the disclosed embodiments and by reference to the drawings and claims.

[0025] Exemplary embodiments of the present invention provide a method for maintaining the layout of a free-form electronic document, by moving objects after collisions so that the user can clearly see the individual objects while working with the electronic document.

[0026] FIG. 1 illustrates an exemplary operating environment for implementating the present invention. The exemplary operating environment includes a general-purpose computing device in the form of a conventional personal computer 120. Generally, the personal computer 120 includes a processing unit 121, a system memory 122, and a system bus 123 that couples system components including the system memory 122 to the processing unit 121. The system bus 123 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any bus architecture. The system memory includes a read-only memory (ROM) 124 and a random access memory (RAM) 125. A basic input/output system (BIOS) 126, containing the basic routines that help to transfer information between elements within personal computer 120, such as during start-up, is stored in ROM 124.

[0027] Personal computer 120 further includes a hard disk drive 127 for reading from and writing to a hard disk, not shown, a magnetic disk drive 128 for reading from or writing to a removable magnetic disk 129, and an optical disk drive 130 for reading from or writing to a removable optical disk 131 such as a CD-ROM or other optical media. Hard disk drive 127, magnetic disk drive 128, and optical disk drive 130 are connected to system bus 123 by a hard disk drive interface 132, a magnetic disk drive interface 133, and an optical disk drive interface 134, respectively. Although the exemplary environment described herein employs hard disk 127, removable magnetic disk 129, and removable optical disk 131, it should be appreciated by those skilled in the art that other types of computer readable media that can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, RAMs, ROMs, and the like, may also be used in the exemplary operating environment. The drives and their associated computer-readable media provide nonvolatile storage of computer-executable instructions, data structures, program modules, and other data for personal computer 120.

[0028] A number of program modules may be stored on hard disk 127, magnetic disk 129, optical disk 131, ROM 124, or RAM 125, including an operating system 135, a document editor 136, a collision resolution module 138, and an electronic document 139. Program modules include routines, sub-routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Aspects of the present invention may be implemented in the form of a document editor 136 that can operate in concert with the collision resolution module 138 to edit the electronic document 139. The document editor 136 generally comprises computer-executable instructions for creating and editing electronic documents. The collision resolution module 138 is generally accessible to the document editor 136, but also can be implemented as an integral part of the document editor.

[0029] A user may enter commands and information into personal computer 120 through input devices, such as a keyboard 140 and a pointing device 142. Pointing devices may include a mouse, a trackball, and an electronic pen that can be used in conjunction with an electronic tablet. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to processing unit 122 through a serial port interface 146 that is coupled to the system bus 123, but may be connected by other interfaces, such as a parallel port, game port, a universal serial bus (USB), or the like. A display device 147 may also be connected to system bus 123 via an interface, such as a video adapter 148. In addition to the monitor, personal computers typically include other peripheral output devices (not shown), such as speakers and printers.

[0030] The personal computer 120 may operate in a networked environment using logical connections to one or more remote computers 149. Remote computer 149 may be another personal computer, a server, a client, a router, a network PC, a peer device, or other common network node. While a remote computer 149 typically includes many or all of the elements described above relative to the personal computer 120, only a memory storage device 150 has been illustrated in FIG. 1. The logical connections depicted in FIG. 1 include a local area network (LAN) 151 and a wide area network (WAN) 152. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.

[0031] When used in a LAN networking environment, the personal computer 120 is often connected to the local area network 151 through a network interface or adapter 153. When used in a WAN networking environment, the personal computer 120 typically includes a modem 154 or other means for establishing communications over WAN 152, such as the Internet. Modem 154, which may be internal or external, is connected to system bus 123 via serial port interface 146. In a networked environment, program modules depicted relative to personal computer 120, or portions thereof, may be stored in the remote memory storage device 150. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0032] Moreover, those skilled in the art will appreciate that the present invention may be implemented in other computer system configurations, including hand-held devices (PDAs), multiprocessor systems, microprocessor based or programmable consumer electronics, network person computers, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0033] FIG. 2 is a block diagram depicting the primary functional components of an exemplary free-form document editor and related input devices. Specifically, FIG. 2 depicts typical hardware and software components used in operating an exemplary embodiment of the invention in the context of a free-form document editor. Conventional input devices are represented by the keyboard 260 and the pointing device 265 (e.g., mouse, pen and tablet). A user can enter commands and information into the computer 20 using the pointer device or another input devices. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, and scanner. A conventional output device, such as monitor 270, is used to display information from the computer 20. Other output devices (not shown) can include a printer or speaker. Other hardware components shown in FIG. 2 include an electronic tablet 250 and an accompanying stylus 255. The tablet 250 and stylus 255 are used to input handwriting strokes that can be converted to data, referred to as electronic ink. The electronic ink may be incorporated into an electronic document 220 and may be displayed on either the electronic tablet 250, the monitor 270, or both. Although the electronic tablet 250 and the monitor 270 are illustrated as being distinct, in an exemplary embodiment of the present invention, they can be integrated into a single component. The joint tablet/monitor component has the ability to display information and receive input from the stylus 255.

[0034] In the representative architecture 200 all of the hardware components are coupled to an ink processing software module 225. The ink-processing module 225 is operable to receive data from the electronic tablet 250 and to render that data as electronic ink. In one embodiment of the present invention, the ink-processing module 225 can be a collection of software modules that perform different tasks for rendering handwriting strokes as electronic ink. For example, the stylus and ink module 228 can receive data describing the positions and angles of the stylus for a series of handwriting strokes. The stylus and ink module 228 can interpret the data for rendering electronic ink. Other software modules, such as a gesture recognizer 230 and word recognizer 235 can be designed to identify certain handwriting strokes and assign them a particular significance. For example, certain gestures such as a cross-out may be recognized and associated with other editing processes. The ink-processing module 225 can also include an erasing functions module 240 for removing electronic ink that has been previously rendered.

[0035] Although ink processing modules are known in the art and necessary for an electronic tablet to function, a novel document editing module has been developed by the Microsoft Corporation of Redmond, Wash. that capitalizes on the benefits of handwriting processing technology. The module (i.e., document editor) is a free-form document editor that leverages the functionality of electronic handwriting technology to enable more effective and efficient note taking. Specifically, document editing module 205 facilitates manipulating electronic ink so that a user can create and modify an electronic document 220 with greater ease and sophistication. The document editing module 205 typically comprises a collection of software modules for controlling and manipulating electronic ink rendered on the monitor 270. For example, a parsing module 210 can be used to identify handwriting strokes that are selected by the user for editing. Selected strokes may by highlighted or shaded to assist the user in identifying which strokes are to be edited. A classifier module 215 can identify certain handwriting strokes as being part of a word or drawing and may designate document objects accordingly. Software modules such as the layout module 216 and the collision resolution module 217 can be designed to control how objects move about a page.

[0036] Exemplary embodiments of the present invention relate most directly to managing the layout in an electronic document by resolving object collisions. The functionality of embodiments may be performed by a collision resolution module 217 or may be fully integrated into the operation of the document editing module 205. FIGS. 1 and 2 provide a general overview of the environments in which the inventors contemplate exemplary embodiments of the present invention will be used.

[0037] FIG. 3 presents an exemplary embodiment of a free-form electronic document screen 300 with document objects. A document object 302 is an outline object and includes an outline heading 310 and outline text 312. Below the outline object is a drawing object 306 and beneath that object is a text object 308 containing a heading 314 and a list of items 316. To the right of the outline object 302 is a handwriting object 304. The handwriting object 304 illustrates an object that may have been added to the document using a stylus and electronic tablet. Each of these four objects can be moved about the page, resized, or edited, or a new object may be added to the page. FIG. 3 depicts document objects with rectangular boundaries. One skilled in the art would appreciate that object boundaries may also be defined as co-extensive with the object contents.

[0038] FIG. 4 presents an exemplary free-form electronic document screen 400 with an outline object 402, a handwriting object 404, a drawing object 406, and a text object 408. This screen is similar to the screen depicted in FIG. 3. However, FIG. 4 shows that the outline object 402 has been modified; text has been added to the outline object, for example, by using a keyboard. When text was added to the outline, the outline object 402 grew and overlapped the drawing object 406. This overlap is depicted by the cross-hatched region 410. This overlap is the result of a document collision. One skilled in the art would appreciate that collision rules other than overlapping can be used, e.g., when one object comes within a small distance of another.

[0039] In an exemplary embodiment of the present invention a determination is made every time a document page is modified as to whether a document object collision has occurred. Modifications include editing an object (illustrated in FIG. 4), resizing an object, moving an object, or adding an object. If a document object collision is determined, then the resolution process is triggered.

[0040] The cross-hatched region 410 representing a document object overlap in FIG. 4 has a width “X” 414 and a height “Y” 412. In an exemplary embodiment of the present invention the shortest distance needed to move the drawing object 406 to resolve the collision is determined. In an embodiment of the present invention, the collision is resolved in real time and the object hit by the edited, moved, resized, or added object is moved the shortest distance. In the collision illustrated in FIG. 4, the height 412 of the collision region 410 is less than the width 414 of the region, so the shortest distance the drawing object 406 can move to resolve the collision is the distance represented by Y. Accordingly, in an exemplary embodiment of the present invention, the object is moved straight down. The example in FIG. 4 depicts a collision between objects whose boundaries are represented by rectangular boxes. One skilled in the art would realize that in another embodiment, an object's boundaries could be defined by the boundaries of the object content. In that case, a collision rule could define a collision as when content of one object overlaps content of another object.

[0041] FIG. 5 depicts an exemplary free-form document screen 500 as it may look after the collision illustrated in FIG. 4 is resolved by an exemplary embodiment of the present invention. The outline object 502, which collided with a drawing object 506 when text was added to the outline object 502, may remain in the same position as when the collision occurs. The drawing object 506 moves straight down the page from its pre-collision location on the page, depicted by the gray shaded box 508, to its new location, depicted by the dash-lined box 506. When the initial collision depicted in FIG. 4 is resolved in this example, the resulting movement caused a secondary collision between the drawing object 506 and the text object 510. The collision is illustrated by the overlap region 512, shown cross-hatched in FIG. 5. An exemplary embodiment of the present invention, after resolving the initial collision, will determine if the resolution resulted in additional collisions. If additional collisions are identified, the additional collisions are resolved. This process of determining and resolving collisions ends when no more collisions exist.

[0042] FIG. 6 depicts an exemplary free-form document screen 600 as it may look after the secondary collision illustrated in FIG. 5 is resolved by an exemplary embodiment of the present invention. The drawing object 606, which collided with a text object 608 after the initial collision is resolved, may remain in the same position as when the secondary collision occurs. The text object 608 may be moved straight down the page from a pre-collision location on the page, depicted by the gray shaded box 610, to a new location, depicted by the dash-lined box 608. The outline object 602 and the handwriting object 604 may not be affected by the collision resolution.

[0043] FIG. 7 presents an exemplary free-form electronic document screen 700 with an outline object 702, a handwriting object 704, a drawing object 706, and a text object 708. FIG. 7 depicts the free-form screen after a collision between drawing object 706 and text object 708. This collision could have occurred, for example, by increasing the size of the drawing object 706. Following a page modification, including resizing an object, the present invention makes a layout pass to identify any collisions. In the example in FIG. 7, the overlap region 710 represents the results of a collision. An exemplary embodiment of the present invention compares the width of the overlap region X 712 with the height of the overlap region Y 714. In the illustration in FIG. 7, X is less than Y, meaning that the shorter distance for resolving the collision is to the side, i.e., moving the text object 708 to the right moves the object the shortest distance while resolving the collision between the drawing object 706 and the test object 708. FIG. 7 depicts document objects with rectangular boundaries. One skilled in the art would appreciate that object boundaries may also be defined as co-extensive with the object contents and a collision rule could define a collision as when content of one object overlaps content of another object.

[0044] FIG. 8 depicts an exemplary free-form electronic screen 800 as it may look after the collision illustrated in FIG. 7 is resolved. The text object 808 is moved straight to the right from its pre-collision location on the page, depicted by the gray shaded box 810, to its new location, depicted by the dash-lined box 808. The outline object 802 and the handwriting object 804 may not be affected by the collision resolution. The exemplary resolution depicted in FIG. 8 shows the resulting location of text object 808 is fully within the electronic screen 800. In an exemplary embodiment of the present invention, if only a section of an object moved to the side following a collision resolution with a width less than one inch remains on the screen, then the object may not be moved to the side. Instead, the object may be moved straight down from its original location. One skilled in the art would appreciate that a variety of rules can be established for managing objects that move partly or completely outside the screen boundaries.

[0045] An exemplary embodiment of the present invention may maintain information on objects' original positions. If a collision resolution operation moves an object, and later editing, moving, or resizing objects opens up the original position of the moved object, i.e., space on the document page originally occupied by the document object is free of any other document object, that object can be returned to its original location. FIG. 9 presents an exemplary free-form electronic document screen 900 illustrating the return of an object to its original position, using the collision resolution depicted in FIG. 8. FIG. 9 shows the screen layout after the drawing object 806 that caused the collision in FIG. 8 is moved or deleted. In this case, the text object 906 returns to its pre-collision location, illustrated by the gray shaded box in FIG. 9.

[0046] FIG. 10 is a flow chart depicting an exemplary method for resolving document object collisions. The method begins at start block 1000 and proceeds to decision block 1002, where the method determines if a layout change is detected, caused by resizing, editing or moving an existing object or adding a new document object. If a layout change is detected, then a layout pass is performed in step 1004 by the document editor. Otherwise, the method branches to end block 1024 and terminates. In decision block 1006, a determination is made as to whether any collision resulted from the layout change. If no collision occurred, the method branches to end block 1024 and terminates. If a collision did result from the layout change, the method moves to step 1008.

[0047] In step 1008, the original location of the object is recorded. Also in step 1008, the distance in the side-to-side direction of the page and the distance in the top-to-bottom direction of the page necessary to move the second object (the object hit as a result of the layout change) to resolve the collision is measured. If the decision step 1010 determines that the side-to-side distance is greater than or equal to the top-to-bottom distance, the object is moved straight down, step 1012. If not, the object is moved straight to the side, step 1014. At decision block 1016, this exemplary embodiment of the present invention determines if, by moving the object to the side, the object has been moved off the screen, e.g., a minimum of one inch in width of the object must be within the screen limits to be considered on the screen. If the resolution of the collision moves the object off the screen, the object is instead moved straight down from its pre-collision location, step 1018. If the object remains on the screen, the method moves to step 1020, and the object remains moved to the side. After the object has been moved as part of resolving a collision, the method, in step 1022 determines if any other collisions exist, including secondary collisions resulting from moving the object to resolve an initial collision. If additional collisions exist, the process repeats, beginning at step 1008. Otherwise, the method branches to end block 1024 and terminates. In one embodiment, the invention would return a moved object to its original location if the page is subsequently modified and the modification allows the object to return to its original location without causing a new collision.

[0048] In an alternative embodiment of the present invention, the moving an object to resolve a collision may not be desired. This alternative embodiment can best be described by reference to figures. FIG. 11 depicts an exemplary free-form electronic document screen 1100 with four objects, a drawing object 1102, a handwriting object 1104, an outline object 1106, and a text object 1108. These objects can be edited, resized, moved, deleted, or a new object can be added to the page. In this alternative embodiment of the present invention, document objects that are moved on top of outline objects, added to an outline object's void space, or objects manually designated as “not able to collide” are not moved when a collision occurs. In these three cases, the alternative embodiment of the present invention encodes these objects with a designation that the method recognizes as meaning not to move the object if a collision needs to be resolved. One skilled in the art will appreciate that a variety of collision rules can be established for determining whether to move an object following a collision, i.e., designating a document object as “not able to collide.” An outline object 1106 is characterized by a title 1110, multi-level text structure 1112, and void space 1114.

[0049] FIG. 12 depicts the same free-form electronic document screen as depicted in FIG. 11. However, the handwriting object is moved 1204 over on top of the outline object 1206. The handwriting object's original location is represented by the gray shaded area 1210. Once the handwriting object 1204 is moved on top of the outline object 1206, the characteristic of the handwriting object 1204 becomes designated by the alternative embodiment of the present invention as “not able to collide.” If an object is edited, resized, moved, or added and it overlaps the handwriting object 1204, the collision will not be resolved.

[0050] FIG. 13 illustrates a collision that this alternative embodiment of the present invention may not resolve. During document editing, the drawing object 1302 is resized and overlaps the handwriting object 1304, but not the outline object 1306. The overlap region is depicted in the cross-hatched area 1310. Since the handwriting object 1304 had been moved over the outline object 1306 and became designated as “not able to collide,” the alternative embodiment of the present invention will not move the object. However, if the resizing caused the drawing object 1302 to overlap with the outline object 1306, then the collision would be resolved and the outline object 1306 and handwriting object 1304 would move as one object.

[0051] FIG. 14 is a flow chart depicting an exemplary method for resolving document object collisions. The method begins at start block 1400 and proceeds to decision block 1402, where the method determines if a layout change is detected, caused by resizing, editing or moving an existing object or adding a new document object. If a layout change is detected, then a layout pass is performed in step 1404 by the document editor. Otherwise, the method branches to end block 1426 and terminates. In decision block 1406, a determination is made as to whether any collision resulted from the layout change. If no collision occurred, the method branches to end block 1426 and terminates. If a collision did result from the layout change, the method moves to decision block 1408. At decision block 1408, the method determines whether the document object has been designated as not able to collide. If the object has been designated as not able to collide, the method branches to end block 1426 and terminates. Otherwise, the method branches to step 1410.

[0052] In step 1410, the original location of the object is recorded. Also in step 1410, the distance in the side-to-side direction of the page and the distance in the top-to-bottom direction of the page necessary to move the second object (the object hit as a result of the layout change) to resolve the collision is measured. If the decision step 1412 determines that the side-to-side distance is greater than or equal to the top-to-bottom distance, the object is moved straight down, step 1414. If not, the object is moved straight to the side, step 1416. At decision block 1418, this alternative embodiment of the present invention determines if, by moving the object to the side, the object has been moved off the screen, e.g., a minimum of one inch in width of the object must be within the screen limits to be considered on the screen. If the resolution of the collision moves the object off the screen, the object is instead moved straight down from its pre-collision location, step 1420. If the object remains on the screen, the method moves to step 1422, and the object remains moved to the side. After the object has been moved as part of resolving a collision, the method, in step 1424 determines if any other collisions exist, including secondary collisions resulting from moving the object to resolve an initial collision. If additional collisions exist, the process repeats, beginning at step 1408. Otherwise, the method branches to end block 1426 and terminates. In one embodiment, the invention would return a moved object to its original location if the page is subsequently modified and the modification allows the object to return to its original location without causing a new collision.