Generation and distribution of annotation overlays of digital X-ray images for security systems

[00002] This application claims the benefit of U.S. Provisional Patent Application Ser. No. 60/220,531 filed on Jul. 24, 2000.

[00003] The present invention relates to an image display storage and retrieval system. More specifically, the invention relates to x-ray image annotation and transmission over a network for security systems.

[00004] Security at high traffic installations such as airport terminals typically requires inspection of parcels passing certain checkpoints. Travelers generally prefer to avoid lengthy delays imposed by such inspections. Hence, in order to expedite inspection within the shortest period of time and with a minimum of privacy infringement, security systems have been established to screen the parcels, by radiating x-rays through the parcels to a detector to produce an image. This image can be viewed by a security agent on an image display device, and if found innocuous, will result in the package being passed on for delivery to the traveler or the transport. If the image suggests further inspection is required, the package may be isolated from others and physically searched in more detail to ascertain and assess its contents.

[00005] Carry-on parcels to be screened at an airport are usually inspected via an x-ray screening system. Passengers' place parcels onto an input conveyor belt. The belt transports the parcels through the inspection system, which x-rays the parcels and presents images to the x-ray system operator displaying the contents of the parcels on a video monitor. When the x-ray system operator observes a potentially prohibited item appear on the monitor, he may provide a signal to a search operator. This signal may constitute an oral statement, or be exhibited as an audio or visual alarm indicator implemented electronically via a push-button that provides a delayed indication when the parcel in question reaches the exit of the x-ray system. The identity of the parcel to be further examined is sometimes not clearly indicated. On occasion, the search operator examines a parcel that does not conform to the parcel the x-ray system operator intended when the alarm was initiated. Additionally, the search operator has no indication of what is in the parcel to be searched, or what aroused the x-ray operator's suspicion. On occasion, this may permit a parcel that should be physically searched for a particular potential item of contraband to escape the physical inspection process.

[00006] One solution proposed in the past includes transmitting an integrated image from multiple x-ray emitters to record on a video disk and transfer to a monitor for continuous display, from which it can be sent by a monitor operator to a parcel sorter who diverts the parcel to a separate inspection desk for further examination. A more effective method to properly identifying parcels to be segregated from others for more detailed inspection is needed in the security industry.

[00007] An image display storage and retrieval system provides a mechanism to transmit x-ray images of parcels to one or more remote workstations. The images may be annotated at these workstations to specifically identify articles to be targeted for more thorough investigation. The image display storage and retrieval system comprises an x-ray image source to illuminate a parcel with x-rays, scan the received x-ray pattern produced from the x-rays passing through the parcel, and digitize the x-ray image of the parcel, an initial screening station where images are initially received and may be annotated, an inspection station where the images (with or without annotation) and parcels to be inspected are received, an optional parcel path switch to direct parcels to either a clearance station or an inspection station, a data storage and retrieval device to record and retrieve images, a data processor to receive the images and a data network connecting the initial screening station, the inspection station, the optional path switch, the data storage and retrieval device and the data processor so as to enable the exchange of data.

[00017] Embodiments of the present invention are described herein in the context of a Search Workstation Display. Those of ordinary skill in the art will realize that the following detailed description of the present invention is illustrative only and is not intended to be in any way limiting. Other embodiments of the present invention will readily suggest themselves to such skilled persons having the benefit of this disclosure. Reference will now be made in detail to implementations of the present invention as illustrated in the accompanying drawings. The same reference indicators will be used throughout the drawings and the following detailed description to refer to the same or like parts.

[00018] In the interest of clarity, not all of the routine features of the implementations described herein are shown and described. It will, of course, be appreciated that in the development of any such actual implementation, numerous implementation-specific decisions must be made in order to achieve the developer's specific goals, such as compliance with application- and business-related constraints, and that these specific goals will vary from one implementation to another and from one developer to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skill in the art having the benefit of this disclosure.

[00019] In accordance with the present invention, the components, process steps, and/or data structures may be implemented using various types of operating systems, computing platforms, computer programs, and/or general-purpose machines. In addition, those of ordinary skill in the art will recognize that devices of a less general purpose nature, such as hardwired devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or the like, may also be used without departing from the scope and spirit of the inventive concepts disclosed herein.

[00020] In the context of the present invention, the term “network” includes local area networks, wide area networks, metropolitan area networks, the Internet, cable television systems supporting bi-directional data communication, telephone systems, wireless data telecommunications systems, fiber optic networks, ATM (asynchronous transfer mode) networks, frame relay networks, satellite data telecommunications networks, and the like. Such networks are well known in the art and consequently are not further described herein.

[00021] The present invention is directed to distributing digital images of x-ray scans to one or more workstations in order to facilitate identification of possible contraband. The image display storage and retrieval system comprises an x-ray image source to illuminate a parcel with x-rays, scan the received x-ray pattern produced from the x-rays passing through the parcel, and digitize the x-ray image of the parcel. The original x-ray image is created by a scanner in the form of a digital image and stored digitally on a data storage and retrieval device. A data network enables one or more stations to access and annotate the digital image. In sending an annotated image to an inspection station, visual clues identifying the articles attracting attention or suspicion may be appended to the image in digital form. In addition, the annotated image can be sent in more than one direction. Instead of being restricted to image transmission from a scanning station to a searching station; the digital image (in original form or with appended annotation reference marks) may be sent from any station to any other station on the network.

[00022] The present invention provides a search operator with the scanned x-ray image that the x-ray operator may have observed at the time of initiating the alarm for a search response. A digital scan system creates a digital x-ray image, which can be annotated by an operator and be transmitted over a network to enable multiple search workstations to access the digital image. In addition, the system may function in conjunction with the Threat Image Projection (TIP) system, so that an archived image featuring particular contraband for training identification may be substituted on a selected basis. FIG. 1 illustrates a diagram of a presently preferred embodiment of the present invention. A security system 10 for screening and inspection is depicted as equipment support for one or more x-ray screen operators 12 and one or more search operators 14. Several parcels 16 are located at the loading station 18. A parcel 16′ enters a x-ray scanning chamber 20, which may include a x-ray emitter 22 and a camera 22′ to register a x-ray image, which may be scanned into a digital form. This digital x-ray image may be transmitted via the network server 24 via network connections 24′ to a computer 26 capable of data processing and recorded on a storage and retrieval device 28. Software to enable annotation overlay of images, message text composition, image compression and decompression, and other related functions may be provided by a storage medium 28′. This software may be loaded and executed by the computer 26 after being loaded by storage medium 28. Additional software for Intranet transmission of images between stations may be made available on a storage medium 28″ to the network server 24. The storage retrieval device 28 for reading and/or writing data may use magnetic or optical media 28′, such as for example a hard-disk drive, magnetic tape or DVD for permanent archival storage of image data or software, or CD-ROM or flash memory for interactive retrieval of image data or rapid access to software, such as decompression applications.

[00023] The network may include software to perform as a distributed system so as to render the file transfer automatic to the workstation operators. The transmission may incorporate a local area network (LAN) with an arbitration mechanism to control message sequencing and may be centralized or distributed. One example of a decentralized bus-based broadcast system is the IEEE 802.03 standard called Ethernet™. In one preferred embodiment of the present invention, the use of the network server 24 as a central controller, the network connections 24′ may comprise twisted pair wire such as 10Base-T. An alternate embodiment may use cable such as 10Base-2 (thin Ethernet) or 10Base-F (fiber optic). Another preferred embodiment of the present invention may incorporate a token ring under the IEEE 802.5 standard used in digital applications. The protocol for such a system may include Transmission Control Protocol (TCP) with an Internet Protocol (IP) packet format, typically expressed as a TCP-IP network. An alternate LAN package used for personal computers (PC) may be Novell Netware® in which each PC functions as a client. Another alternate LAN package used between Apple® platforms may be IEEE 1394 known as FireWire™. Alternatively, a wireless network system may be used in which network connections 24′ may represent infrared signal transmitted from one workstation to the network server 24 and vice versa. These examples are not considered limiting, and many other network arrangements may be considered without departing from the present invention.

[00024] The digital x-ray scan of parcel 16′ is preferably stored in an addressed first memory location 30 on the recorder 28. The network server 24 causes the network connections 24′ to transmit the digital scanned x-ray image to an initial screening workstation 32. This transmission may be performed contemporaneously with the capture of the scanned x-ray image. The screen monitor 34 at screening terminal or workstation 32 displays the digital scanned x-ray image to the screen operator 12. The screening workstation 32 may include one or more devices for annotating the x-ray image to produce a first annotated image, including a keyboard 36, touch screen 36′, and/or cursor tracing device 36″ (e.g., trackpad, mouse, joystick). This first annotated image may be composed of the digital scanned x-ray image in original form plus a first overlay image featuring the annotation supplied by the screen operator 12. The first annotated image may constitute a single image file or form the first overlay combined with a data call to the digital scanned x-ray image, such as by accessing an alias to that image. The latter arrangement separating the overlay and the original image may consume less data storage than a combined image storage. This is due to the generally much smaller storage requirements for an annotation layer than the digital scanned x-ray image due to the larger quantity of image data registered on the digital scanned x-ray image, even when compressed using glossy image compression techniques well known in the art. In addition, the screening workstation 32 may include an alarm initiator 38, such as a push-button or toggle switch, which may be activated by the screen operator 12 sending an alarm signal to the computer 26 when a parcel 16′ is suspected to contain contraband. The alarm initiator 38 may be set to off, on and reset positions. The first annotated image is transmitted over the network connections 24′ controlled via the network server 24 to the computer 26 to be stored in digital form in recorder 28 in an addressed second memory location 40. The screen operator 12 may maintain the scanned x-ray image of parcel 16′ on the screen monitor 34, or may clear the digital scanned x-ray image or the first annotated x-ray image by means of a button or toggle or similar-on-off device to enable receipt of a scanned x-ray image from a subsequent parcel or from the archive.

[00025] After passing through the x-ray chamber 20, the scanned parcel 42 may be disposed along a conveyor 44 or other similar transport mechanism to a path switch or diverter 46. In the absence of an alarm signal from the screen operator 12, a parcel 42′ may be sent by default to a clearance station 48 to be retrieved by the traveler 50 or loaded onto an awaiting transport 52. When the screen operator 12 sends an alarm signal from alarm initiator 38 transmitted across the network connectors 24′, the scanned parcel 42″ may be diverted to an available inspection station 54 or 56 for physical examination by a search operator 14. The computer 26 may impose a delay across the network connectors 24′ of alarm signal at the inspection station 56 until the scanned parcel 42″ reaches the inspection station 56.

[00026] The annotated x-ray image or annotation overlay image may be stored in second memory location 44 from the recorder 28. From the screening workstation or the memory storage, this image may be transmitted over the network connections 24′ controlled via the network server 24 to the inspection station 56 with a search monitor 58 observable to the search operator 14. The inspection station 56 with a search monitor 58 observable to the search operator 14. The inspection station 56 may also include one or more devices for annotating the first annotated or scanned x-ray image to produce a second annotated image, including a keyboard 62, touch screen 62′, and/or cursor tracking device 62″ (e.g., trackpad, mouse, joystick). The second annotated image may constitute a single image file or form the second and/or first overlays combined with a data call to the digital scanned x-ray image, such as by accessing an alias to that image. The latter arrangement separating the overlays and the original image may consume less data storage than a combined image storage. In addition, the search operator 14 may have access to an alarm indicator 64 at the inspection station 56, such as a colored light switched on and/or an audio tone, to receive an alarm signal sent from the screen operator 12 via the alarm initiator 38 across the network connectors 24′. The alarm indicator 64 may include reset position and an acknowledgement signal to alarm initiator 38 via the network connectors 24′.

[00027] Once the parcel 42″ has reached the inspection station 56, the search operator 14 may then open the parcel 42″ and inspect its contents to determine whether suspected contraband 66 can be observed and located. The search operator 14 may annotate the digital scanned x-ray image or the first annotated image to produce a second annotated image describing the suspected contraband 66. The second annotated image may be transmitted over the network connectors 24′ to the computer 26 to be stored in digital form I recorder 28 in an addressed third memory location 68 for further retrieval and evaluation regarding legal consequences. This second annotated image may be composed of the digital scanned x-ray image in original form plus a second overlay image featuring the annotation supplied by the search operator 14 and/or the first overlay image. The search operator 14 may then clear the displayed annotated x-ray image from the search monitor 58 by means of a button or toggle or similar on-off device to enable receipt of a subsequent image, whether from archive or from the screen operator 12. The second annotated image may be send by the search operator to the screening operator as an acknowledgement of the attention drawn to the parcel identified for examination.

[00028] FIGS. 2A through 2F illustrate an example of image annotation. The digital scanned x-ray image shown on screen monitor 34 is illustrated as a scanned x-ray image 70 may be displayed on a screening video monitor in FIG. 2A. A duplicate scanned x-ray image 72 may be displayed on a search video monitor in FIG. 2B. Within the scanned x-ray image 70 one may observe the outline of a suitcase 78 with the handle 80 on the bottom side. The contents displayed by the scanned x-ray image 70 include what appear to be a comb 82, a tube of toothpaste 84, a toothbrush 86, a possible handgun 88 and an unidentified item 90. This scanned x-ray image 70 may be transmitted over the network, stored in the data storage device, and transmitted via the retrieval device to the screening workstation or console.

[00029] The screening operator may proceed to annotate the scanned x-ray image 70 which when displayed independently of the scanned x-ray image may be illustrated in FIG. 2C as an annotation overlay layer 92. When combined with the scanned x-ray image 70, the screening operator may produce a screen annotated image 94 in FIG. 2D. An image annotation software executable program, such as Adobe® Photoshop® may be may be used to enable an operator to create illustrating features such as lines on the workstation screen as an overlay image which when superimposed over the scanned x-ray image 70 to produce the screen annotated image 94. A point-tracking device, such as a touch screen, trackpad, joystick or mouse, may be used to trace a screen identifying mark 96 to specify the article in the suitcase of interest using the image annotation software. In this example, the screen identifying mark 96 is drawn around the possible handgun 88. A keyboard entry may enable the screening operator to compose a text-based instruction 98 on the overlay image to the search operator. The screen annotation layer 92 may be stored in the data storage device and transmitted over the network to the search monitor as an overlay to be superimposed over the duplicate x-ray image 72. The screen annotation overlay 92 may then be transmitted over the network, stored and accessed in the data storage and retrieval device to the search workstation to be superimposed on the duplicate scanned x-ray image 72 displayed on the search monitor. Alternatively, a screen annotated image 94 featuring the scanned x-ray image 70 with the annotation layer 92 may be transmitted over the network.

[00030] An overlay image may be stored a separate file distinguished from the scanned x-ray image 70, whereas the screen annotated image 94 might represent a combined image displayed on a monitor. In one embodiment, the overlay image might undergo GIF compression as a graphic to reduce data storage requirements. Additionally, or alternatively, the scanned x-ray image 70 might undergo JPEG compression as a photograph to reduce data storage requirements. The overlay image and the scanned x-ray image 70 might be hyperlinked to URL links for retrieval within a local area network (LAN) or an Intranet system.

[00031] The search operator may physically examine the contents of the depicted suitcase 78. From the screen identifying mark 96 and/or the instruction 98, the search operator may be made aware of the reasons why the screen operator did not permit this suitcase 78 to proceed to the clearance station. The search operator may proceed to annotate the screen annotated image 94 displayed to produce a search annotation overlay 100 in FIG. 2E. The search annotation overlay 100 may be superimposed over the screen annotated image 94 yielding a search annotated image 102 in FIG. 2F. Once the search operator physically confirms that the possible handgun 88 is actually a firearm prohibited in the security area, a keyboard entry may be used to compose a text-based response 104 to the instruction 98. Furthermore, the search operator may on further investigation, discover the unidentified item 90 constitutes a switchblade knife, also prohibited, and may proceed to encircle that item in the search annotation 100 by a search identifying mark 106. The search annotated layer 100 may then be transmitted over the network and stored I the data storage and retrieval device for future archive.

[00032] The digital scanned x-ray image, which may or may not be annotated with an annotation layer by the x-ray operator or the screening operator, may be transmitted over the network to one or more search operators and may be synchronized with the alarm signal to authorize a physical examination or search of the parcel identified by the alarm. The digital scanned x-ray image with or without an overlay image may be presented on a search monitor as a search display image until the search display image is cleared from the monitor screen by a switch actuated by the search operator. The search operator may verify that the parcel being searched matches the search display image on the search monitor. The search display image of the digital scanned x-ray image on the search monitor will facilitate the search operator in examining the parcel. Meantime, x-ray inspection of subsequent parcels may continue uninterrupted by the x-ray operator with the x-ray screening system. If one or more subsequent parcels need to be physically examined before examination of the first parcel is completed, the x-ray screening system may be paused until the search station may be available or an additional search station becomes available.

[00033] The scanned x-ray image may be produced as or converted to a digital image and stored on a computer or an alternate electromagnetic, optical or electronic storage device. This storage device may save the image and/or transmit the image via the network to another device such as a computer for the search workstation platform. The image may be digitized, encrypted, annotated, reassembled and/or stored as a packet for data protection. By recording digital data rather than analog data, the expense of memory usage may be reduced and versatility in annotating images may be improved, thus facilitating the screen operator's task of identifying packages deemed suspect and warranting closer scrutiny. The particular image to be displayed on either the x-ray screen monitor or the search monitor may be selected. The status of the workstation platform may also be transmitted over the network, such as under TCP/IP other local area network (LAN) protocol.

[00034] Multiple search workstation platforms might be connected to the network in order to access the digital images available to multiple authorized personnel to expedite resolution of whether the suspect parcel contains contraband. If a digital image of a particular parcel being projected at the time of the search command is provided, the x-ray screening system may be paused in order to provide sufficient time to relocate that parcel, and prompt the x-ray operator for a clear or search response. This interruption command of the x-ray screening system may also be expanded to hold a plurality of parcels in queue, with resumption based on one or more selected criteria. Furthermore, the receipt and transmission of images may be controlled from a central site independent of the x-ray screen station and/or the inspection station.

[00035] While embodiments and applications of this invention have been shown and described, it would be apparent to those skilled in the art having the benefit of this disclosure that many more modifications than mentioned above are possible without departing from the inventive concepts herein. The invention, therefore, is not to be restricted except in the spirit of the appended claims.