Multi-modality vertebra recognition

[0001] The present disclosure relates to recognition of vertebra structures in radiology images, and more particularly to systems, methods and computer program products to recognize vertebra structures from multiple modalities and using arbitrary views.

[0002] The statements in this section merely provide background information related to the disclosure and may not constitute prior art.

[0003] Automatic spine recognition which supports quantitative measurement is essential in numerous spine related applications in orthopedics, neurology, and oncology. The task of automatic spine recognition is to extract the set of numerical parameters that can uniquely determine the global structure of the spine and certain local structures of the vertebrae. Currently, spine recognition is often simplified as vertebra detection, which extracts the locations and labels of the vertebrae in input images.

[0004] Spine recognition is a challenging problem in spine image analysis. In the present application spine recognition is accomplished, in part, by using image appearances. Image appearance is defined as a set of geometric parameters that determine the local and global spine structures. The parameters obtained from spine recognition provides a unified geometry model that can be shared among spine structures from different modalities, different image views, and different formats. The appearance parameters for a spine can be used in quantitative measurement systems for diagnostic purposes and can be stored/retrieved by medical PACS systems. The main difficulties of spine recognition arise from the high variability of image appearance due to modality differences or shape deformations: 1) Multiple modalities. The image resolution, contrast, and appearance for the same spine structure are very different when it is exposed to CT, or T1/T2 weighted MR images. 2) High repetition. The appearances of vertebrae and intervertebral discs are highly repetitive which often leads to mismatching/misrecognition. 3) Various poses. The vertebrae sizes and orientations are highly diverse in pathological data therefore regular detectors such as appearance detectors are insufficient to match all vertebra. 4) Complex shape composition. The spine is composed of local vertebrae and the compositional shape can be highly twisted and cannot be described by simple geometric models like curves and surfaces. Therefore, current methods of recognizing local vertebral structures and global spine shapes are often separately done by spine detection and spine shape matching techniques.

[0005] In view of the above, there is a need for systems, methods, and computer program products perform automatic vertebra recognition to identify the global spine and local vertebra structural information such as spine shape, vertebra location and pose. The above-mentioned needs are addressed by the subject matter described herein and will be understood in the following specification.

[0006] According to one aspect of the present disclosure, a system that allows automatic vertebra recognition to identify the global spine and local vertebra structural information such as spine shape, vertebra location and pose is provided.

[0007] According to another aspect of the present disclosure, a method that allows automatic vertebra recognition to identify the global spine and local vertebra structural information such as spine shape, vertebra location and pose is provided.

[0008] This summary briefly describes aspects of the subject matter described below in the Detailed Description, and is not intended to be used to limit the scope of the subject matter described in the present disclosure.

[0015] In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific examples that may be practiced. These examples are described in sufficient detail to enable one skilled in the art to practice the subject matter, and it is to be understood that other examples may be utilized and that logical, mechanical, electrical and other changes may be made without departing from the scope of the subject matter of this disclosure. The following detailed description is, therefore, provided to describe an exemplary implementation and not to be taken as limiting on the scope of the subject matter described in this disclosure. Certain features from different aspects of the following description may be combined to form yet new aspects of the subject matter discussed below.

[0016] When introducing elements of various embodiments of the present disclosure, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements.

[0017] Aspects disclosed and described herein enable automatic vertebra recognition to identify the global spine and local vertebra structural information such as spine shape, vertebra location and pose. In the present application, a vertebra recognition method using a 3D deformable hierarchical model (DHM) to achieve cross-modality local vertebra location and pose identification with vertebra labeling, and global 3D spine shape recovery is considered.

[0018] Other aspects, such as those discussed in the following and others as can be appreciated by one having ordinary skill in the art upon reading the enclosed description, are also possible.

[0019] Health information, also referred to as healthcare information and/or healthcare data, relates to information generated and/or used by a healthcare entity. Health information can be information associated with health of one or more patients, for example. Health information may include protected health information (PHI), as outlined in the Health Insurance Portability and Accountability Act (HIPAA), which is identifiable as associated with a particular patient and is protected from unauthorized disclosure. Health information can be organized as internal information and external information. Internal information includes patient encounter information (e.g., patient-specific data, aggregate data, comparative data, etc.) and general healthcare operations information, etc. External information includes comparative data, expert and/or knowledge-based data, etc. Information can have both a clinical (e.g., diagnosis, treatment, prevention, etc.) and administrative (e.g., scheduling, billing, management, etc.) purpose.

[0020] Institutions, such as healthcare institutions, having complex network support environments and sometimes chaotically driven process flows utilize secure handling and safeguarding of the flow of sensitive information (e.g., personal privacy). A need for secure handling and safeguarding of information increases as a demand for flexibility, volume, and speed of exchange of such information grows. For example, healthcare institutions provide enhanced control and safeguarding of the exchange and storage of sensitive patient PHI and employee information between diverse locations to improve hospital operational efficiency in an operational environment typically having a chaotic-driven demand by patients for hospital services. In certain examples, patient identifying information can be masked or even stripped from certain data depending upon where the data is stored and who has access to that data. In some examples, PHI that has been “de-identified” can be re-identified based on a key and/or other encoder/decoder.

[0021] A healthcare information technology infrastructure can be adapted to service multiple business interests while providing clinical information and services. Such an infrastructure may include a centralized capability including, for example, a data repository, reporting, discreet data exchange/connectivity, “smart” algorithms, personalization/consumer decision support, etc. This centralized capability provides information and functionality to a plurality of users including medical devices, electronic records, access portals, pay for performance (P4P), chronic disease models, and clinical health information exchange/regional health information organization (HIE/RHIO), and/or enterprise pharmaceutical studies, home health, for example.

[0022] Interconnection of multiple data sources helps enable an engagement of all relevant members of a patient's care team and helps improve an administrative and management burden on the patient for managing his or her care. Particularly, interconnecting the patient's electronic medical record and/or other medical data can help improve patient care and management of patient information. Furthermore, patient care compliance is facilitated by providing tools that automatically adapt to the specific and changing health conditions of the patient and provide comprehensive education and compliance tools to drive positive health outcomes.

[0023] In certain examples, healthcare information can be distributed among multiple applications using a variety of database and storage technologies and data formats. To provide a common interface and access to data residing across these applications, a connectivity framework (CF) can be provided which leverages common data and service models (CDM and CSM) and service oriented technologies, such as an enterprise service bus (ESB) to provide access to the data.

[0024] In certain examples, a variety of user interface frameworks and technologies can be used to build applications for health information systems including, but not limited to, MICROSOFT® ASP.NET, AJAX®, MICROSOFT® Windows Presentation Foundation, GOOGLE® Web Toolkit, MICROSOFT® Silverlight, ADOBE®, and others. Applications can be composed from libraries of information widgets to display multi-content and multi-media information, for example. In addition, the framework enables users to tailor layout of applications and interact with underlying data.

[0025] In certain examples, an advanced Service-Oriented Architecture (SOA) with a modern technology stack helps provide robust interoperability, reliability, and performance. Example SOA includes a three-fold interoperability strategy including a central repository (e.g., a central repository built from Health Level Seven (HL7) transactions), services for working in federated environments, and visual integration with third-party applications. Certain examples provide portable content enabling plug 'n play content exchange among healthcare organizations. A standardized vocabulary using common standards (e.g., LOINC, SNOMED CT, RxNorm, FDB, ICD-9, ICD-10, etc.) is used for interoperability, for example. Certain examples provide an intuitive user interface to help minimize end-user training. Certain examples facilitate user-initiated launching of third-party applications directly from a desktop interface to help provide a seamless workflow by sharing user, patient, and/or other contexts. Certain examples provide real-time (or at least substantially real time assuming some system delay) patient data from one or more information technology (IT) systems and facilitate comparison(s) against evidence-based best practices. Certain examples provide one or more dashboards for specific sets of patients. Dashboard(s) can be based on condition, role, and/or other criteria to indicate variation(s) from a desired practice, for example.

[0026] A. Example Healthcare Information System

[0027] An information system can be defined as an arrangement of information/data, processes, and information technology that interact to collect, process, store, and provide informational output to support delivery of healthcare to one or more patients. Information technology includes computer technology (e.g., hardware and software) along with data and telecommunications technology (e.g., data, image, and/or voice network, etc.).

[0028] Turning now to the figures, FIG. 1 shows a block diagram of an example healthcare-focused information system 100. Example system 100 can be configured to implement a variety of systems and processes including image storage (e.g., picture archiving and communication system (PACS), etc.), image processing and/or analysis, radiology reporting and/or review (e.g., radiology information system (RIS), etc.), computerized provider order entry (CPOE) system, clinical decision support, patient monitoring, population health management (e.g., population health management system (PHMS), health information exchange (HIE), etc.), healthcare data analytics, cloud-based image sharing, electronic medical record (e.g., electronic medical record system (EMR), electronic health record system (EHR), electronic patient record (EPR), personal health record system (PHR), etc.), and/or other health information system (e.g., clinical information system (CIS), hospital information system (HIS), patient data management system (PDMS), laboratory information system (LIS), cardiovascular information system (CVIS), etc.

[0029] As illustrated in FIG. 1, the example information system 100 includes an input 110, an output 120, a processor 130, a memory 140, and a communication interface 150. The components of example system 100 can be integrated in one device or distributed over two or more devices.

[0030] Example input 110 may include a keyboard, a touch-screen, a mouse, a trackball, a track pad, optical barcode recognition, voice command, etc. or combination thereof used to communicate an instruction or data to system 100. Example input 110 may include an interface between systems, between user(s) and system 100, etc.

[0031] Example output 120 can provide a display generated by processor 130 for visual illustration on a monitor or the like. The display can be in the form of a network interface or graphic user interface (GUI) to exchange data, instructions, or illustrations on a computing device via communication interface 150, for example. Example output 120 may include a monitor (e.g., liquid crystal display (LCD), plasma display, cathode ray tube (CRT), etc.), light emitting diodes (LEDs), a touch-screen, a printer, a speaker, or other conventional display device or combination thereof.

[0032] Example processor 130 includes hardware and/or software configuring the hardware to execute one or more tasks and/or implement a particular system configuration. Example processor 130 processes data received at input 110 and generates a result that can be provided to one or more of output 120, memory 140, and communication interface 150. For example, example processor 130 can take user annotation provided via input 110 with respect to an image displayed via output 120 and can generate a report associated with the image based on the annotation. As another example, processor 130 can process updated patient information obtained via input 110 to provide an updated patient record to an EMR via communication interface 150.

[0033] Example memory 140 may include a relational database, an object-oriented database, a data dictionary, a clinical data repository, a data warehouse, a data mart, a vendor neutral archive, an enterprise archive, etc. Example memory 140 stores images, patient data, best practices, clinical knowledge, analytics, reports, etc. Example memory 140 can store data and/or instructions for access by the processor 130. In certain examples, memory 140 can be accessible by an external system via the communication interface 150.

[0034] In certain examples, memory 140 stores and controls access to encrypted information, such as patient records, encrypted update-transactions for patient medical records, including usage history, etc. In an example, medical records can be stored without using logic structures specific to medical records. In such a manner, memory 140 is not searchable. For example, a patient's data can be encrypted with a unique patient-owned key at the source of the data. The data is then uploaded to memory 140. Memory 140 does not process or store unencrypted data thus minimizing privacy concerns. The patient's data can be downloaded and decrypted locally with the encryption key.

[0035] For example, memory 140 can be structured according to provider, patient, patient/provider association, and document. Provider information may include, for example, an identifier, a name, and address, a public key, and one or more security categories. Patient information may include, for example, an identifier, a password hash, and an encrypted email address. Patient/provider association information may include a provider identifier, a patient identifier, an encrypted key, and one or more override security categories. Document information may include an identifier, a patient identifier, a clinic identifier, a security category, and encrypted data, for example.

[0036] Example communication interface 150 facilitates transmission of electronic data within and/or among one or more systems. Communication via communication interface 150 can be implemented using one or more protocols. In some examples, communication via communication interface 150 occurs according to one or more standards (e.g., Digital Imaging and Communications in Medicine (DICOM), Health Level Seven (HL7), ANSI X12N, etc.). Example communication interface 150 can be a wired interface (e.g., a data bus, a Universal Serial Bus (USB) connection, etc.) and/or a wireless interface (e.g., radio frequency, infrared, near field communication (NFC), etc.). For example, communication interface 150 may communicate via wired local area network (LAN), wireless LAN, wide area network (WAN), etc. using any past, present, or future communication protocol (e.g., BLUETOOTH™, USB 2.0, USB 3.0, etc.).

[0037] In certain examples, a Web-based portal may be used to facilitate access to information, patient care and/or practice management, etc. Information and/or functionality available via the Web-based portal may include one or more of order entry, laboratory test results review system, patient information, clinical decision support, medication management, scheduling, electronic mail and/or messaging, medical resources, etc. In certain examples, a browser-based interface can serve as a zero footprint, zero download, and/or other universal viewer for a client device.

[0038] In certain examples, the Web-based portal serves as a central interface to access information and applications, for example. Data may be viewed through the Web-based portal or viewer, for example. Additionally, data may be manipulated and propagated using the Web-based portal, for example. Data may be generated, modified, stored and/or used and then communicated to another application or system to be modified, stored and/or used, for example, via the Web-based portal, for example.

[0039] The Web-based portal may be accessible locally (e.g., in an office) and/or remotely (e.g., via the Internet and/or other private network or connection), for example. The Web-based portal may be configured to help or guide a user in accessing data and/or functions to facilitate patient care and practice management, for example. In certain examples, the Web-based portal may be configured according to certain rules, preferences and/or functions, for example. For example, a user may customize the Web portal according to particular desires, preferences and/or requirements.

[0040] B. Example Healthcare Infrastructure

[0041] FIG. 2 shows a block diagram of an example healthcare information infrastructure 200 including one or more subsystems such as the example healthcare-related information system 100 illustrated in FIG. 1. Example healthcare system 200 includes a HIS 204, a RIS 206, a PACS 208, an interface unit 210, a data center 212, and a workstation 214. In the illustrated example, HIS 204, RIS 206, and PACS 208 are housed in a healthcare facility and locally archived. However, in other implementations, HIS 204, RIS 206, and/or PACS 208 may be housed within one or more other suitable locations. In certain implementations, one or more of PACS 208, RIS 206, HIS 204, etc., may be implemented remotely via a thin client and/or downloadable software solution. Furthermore, one or more components of the healthcare system 200 can be combined and/or implemented together. For example, RIS 206 and/or PACS 208 can be integrated with HIS 204; PACS 208 can be integrated with RIS 206; and/or the three example information systems 204, 206, and/or 208 can be integrated together. In other example implementations, healthcare system 200 includes a subset of the illustrated information systems 204, 206, and/or 208. For example, healthcare system 200 may include only one or two of HIS 204, RIS 206, and/or PACS 208. Information (e.g., scheduling, test results, exam image data, observations, diagnosis, etc.) can be entered into HIS 204, RIS 206, and/or PACS 208 by healthcare practitioners (e.g., radiologists, physicians, and/or technicians) and/or administrators before and/or after patient examination.

[0042] The HIS 204 stores medical information such as clinical reports, patient information, and/or administrative information received from, for example, personnel at a hospital, clinic, and/or a physician's office (e.g., an EMR, EHR, PHR, etc.). RIS 206 stores information such as, for example, radiology reports, radiology exam image data, messages, warnings, alerts, patient scheduling information, patient demographic data, patient tracking information, and/or physician and patient status monitors. Additionally, RIS 206 enables exam order entry (e.g., ordering an x-ray of a patient) and image and film tracking (e.g., tracking identities of one or more people that have checked out a film). In some examples, information in RIS 206 is formatted according to the HL-7 (Health Level Seven) clinical communication protocol. In certain examples, a medical exam distributor is located in RIS 206 to facilitate distribution of radiology exams to a radiologist workload for review and management of the exam distribution by, for example, an administrator.

[0043] PACS 208 stores medical images (e.g., x-rays, scans, three-dimensional renderings, etc.) as, for example, digital images in a database or registry. In some examples, the medical images are stored in PACS 208 using the Digital Imaging and Communications in Medicine (DICOM) format. Images are stored in PACS 208 by healthcare practitioners (e.g., imaging technicians, physicians, radiologists) after a medical imaging of a patient and/or are automatically transmitted from medical imaging devices to PACS 208 for storage. In some examples, PACS 208 can also include a display device and/or viewing workstation to enable a healthcare practitioner or provider to communicate with PACS 208.

[0044] The interface unit 210 includes a hospital information system interface connection 216, a radiology information system interface connection 218, a PACS interface connection 220, and a data center interface connection 222. Interface unit 210 facilities communication among HIS 204, RIS 206, PACS 208, and/or data center 212. Interface connections 216, 218, 220, and 222 can be implemented by, for example, a Wide Area Network (WAN) such as a private network or the Internet. Accordingly, interface unit 210 includes one or more communication components such as, for example, an Ethernet device, an asynchronous transfer mode (ATM) device, an 802.11 device, a DSL modem, a cable modem, a cellular modem, etc. In turn, the data center 212 communicates with workstation 214, via a network 224, implemented at a plurality of locations (e.g., a hospital, clinic, doctor's office, other medical office, or terminal, etc.). Network 224 is implemented by, for example, the Internet, an intranet, a private network, a wired or wireless Local Area Network, and/or a wired or wireless Wide Area Network. In some examples, interface unit 210 also includes a broker (e.g., a Mitra Imaging's PACS Broker) to allow medical information and medical images to be transmitted together and stored together.

[0045] Interface unit 210 receives images, medical reports, administrative information, exam workload distribution information, and/or other clinical information from the information systems 204, 206, 208 via the interface connections 216, 218, 220. If necessary (e.g., when different formats of the received information are incompatible), interface unit 210 translates or reformats (e.g., into Structured Query Language (“SQL”) or standard text) the medical information, such as medical reports, to be properly stored at data center 212. The reformatted medical information can be transmitted using a transmission protocol to enable different medical information to share common identification elements, such as a patient name or social security number. Next, interface unit 210 transmits the medical information to data center 212 via data center interface connection 222. Finally, medical information is stored in data center 212 in, for example, the DICOM format, which enables medical images and corresponding medical information to be transmitted and stored together.

[0046] The medical information is later viewable and easily retrievable at workstation 214 (e.g., by their common identification element, such as a patient name or record number). Workstation 214 can be any equipment (e.g., a personal computer) capable of executing software that permits electronic data (e.g., medical reports) and/or electronic medical images (e.g., x-rays, ultrasounds, MRI scans, etc.) to be acquired, stored, or transmitted for viewing and operation. Workstation 214 receives commands and/or other input from a user via, for example, a keyboard, mouse, track ball, microphone, etc. Workstation 214 is capable of implementing a user interface 226 to enable a healthcare practitioner and/or administrator to interact with healthcare system 200. For example, in response to a request from a physician, user interface 226 presents a patient medical history. In other examples, a radiologist is able to retrieve and manage a workload of exams distributed for review to the radiologist via user interface 226. In further examples, an administrator reviews radiologist workloads, exam allocation, and/or operational statistics associated with the distribution of exams via user interface 226. In some examples, the administrator adjusts one or more settings or outcomes via user interface 226.

[0047] Example data center 212 of FIG. 2 is an archive to store information such as images, data, medical reports, and/or, more generally, patient medical records. In addition, data center 212 can also serve as a central conduit to information located at other sources such as, for example, local archives, hospital information systems/radiology information systems (e.g., HIS 204 and/or RIS 206), or medical imaging/storage systems (e.g., PACS 208 and/or connected imaging modalities). That is, the data center 212 can store links or indicators (e.g., identification numbers, patient names, or record numbers) to information. In the illustrated example, data center 212 is managed by an application server provider (ASP) and is located in a centralized location that can be accessed by a plurality of systems and facilities (e.g., hospitals, clinics, doctor's offices, other medical offices, and/or terminals). In some examples, data center 212 can be spatially distant from HIS 204, RIS 206, and/or PACS 208.

[0048] Example data center 212 of FIG. 2 includes a server 228, a database 230, and a record organizer 232. Server 228 receives, processes, and conveys information to and from the components of healthcare system 200. Database 230 stores the medical information described herein and provides access thereto. Example record organizer 232 of FIG. 2 manages patient medical histories, for example. Record organizer 232 can also assist in procedure scheduling, for example.

[0049] Certain examples can be implemented as cloud-based clinical information systems and associated methods of use. An example cloud-based clinical information system enables healthcare entities (e.g., patients, clinicians, sites, groups, communities, and/or other entities) to share information via web-based applications, cloud storage and cloud services. For example, the cloud-based clinical information system may enable a first clinician to securely upload information into the cloud-based clinical information system to allow a second clinician to view and/or download the information via a web application. Thus, for example, the first clinician may upload an x-ray image into the cloud-based clinical information system, and the second clinician may view the x-ray image via a web browser and/or download the x-ray image onto a local information system employed by the second clinician.

[0050] In certain examples, users (e.g., a patient and/or care provider) can access functionality provided by system 200 via a software-as-a-service (SaaS) implementation over a cloud or other computer network, for example. In certain examples, all or part of system 200 can also be provided via platform as a service (PaaS), infrastructure as a service (IaaS), etc. For example, system 200 can be implemented as a cloud-delivered Mobile Computing Integration Platform as a Service. A set of consumer-facing Web-based, mobile, and/or other applications enable users to interact with the PaaS, for example.

[0051] C. Example Methods of Use

[0052] Clinical workflows are typically defined to include one or more steps or actions to be taken in response to one or more events and/or according to a schedule. Events may include receiving a healthcare message associated with one or more aspects of a clinical record, opening a record(s) for new patient(s), receiving a transferred patient, reviewing and reporting on an image, and/or any other instance and/or situation that requires or dictates responsive action or processing. The actions or steps of a clinical workflow may include placing an order for one or more clinical tests, scheduling a procedure, requesting certain information to supplement a received healthcare record, retrieving additional information associated with a patient, providing instructions to a patient and/or a healthcare practitioner associated with the treatment of the patient, radiology image reading, and/or any other action useful in processing healthcare information. The defined clinical workflows may include manual actions or steps to be taken by, for example, an administrator or practitioner, electronic actions or steps to be taken by a system or device, and/or a combination of manual and electronic action(s) or step(s). While one entity of a healthcare enterprise may define a clinical workflow for a certain event in a first manner, a second entity of the healthcare enterprise may define a clinical workflow of that event in a second, different manner. In other words, different healthcare entities may treat or respond to the same event or circumstance in different fashions. Differences in workflow approaches may arise from varying preferences, capabilities, requirements or obligations, standards, protocols, etc. among the different healthcare entities.

[0053] In certain examples, a medical exam conducted on a patient can involve review by a healthcare practitioner, such as a radiologist, to obtain, for example, diagnostic information from the exam. In a hospital setting, medical exams can be ordered for a plurality of patients, all of which require review by an examining practitioner. Each exam has associated attributes, such as a modality, a part of the human body under exam, and/or an exam priority level related to a patient criticality level. Hospital administrators, in managing distribution of exams for review by practitioners, can consider the exam attributes as well as staff availability, staff credentials, and/or institutional factors such as service level agreements and/or overhead costs.

[0054] Additional workflows can be facilitated such as bill processing, revenue cycle mgmt., population health management, patient identity, consent management, etc.

[0055] FIG. 3 depicts an example system 300 for multi-modality vertebrae recognition, according to one aspect of the present disclosure. System 300 includes a computer 302 and a multi-modality vertebrae recognition engine 304 communicatively coupled to computer 302. In this example, computer 302 includes a user interface 306 and a data input (e.g., a keyboard, mouse, microphone, etc.) 308 and multi-modality vertebrae recognition engine 304 includes a processor 310 and a database 312.

[0056] In certain aspects, user interface 306 displays data such as images (e.g., spinal images, radiology images, etc.) and/or images containing recognized vertebrae received from regression segmentation engine 304. In certain aspects, user interface 306 receives commands and/or input from a user 314 via data input 308. In aspects where system 300 is used to segment spinal images, user interface 306 displays a spinal image(s) and user 314 provides an initial input identifying, for example, a location of a vertebra boundaries on the spinal image(s).

[0057] FIG. 4 illustrates a flow diagram of multi-modality vertebrae recognition engine 304 according to one aspect of the present disclosure. In some aspects, multi-modality vertebrae recognition engine 304 generates recognized vertebrae by using a three-stage recognition approach: landmark detection 402, global shape registration 404, and local pose adjustment 406. These stages cover the matching from local to global spine structures. The three stages are implemented by three modules in the hierarchical deformable model 410: the local appearance module 412, global geometry model 414, and local geometry model 416.

[0058] According to one aspect of the present disclosure, the overall workflow can be understood as a three-stage top-down registration. The goal of the registration is: 1) to align the global shape of the spine model, and 2) to align the vertebrae poses with the local image structures around identified landmarks.

[0059] Landmark detection 402 identifies the potential vertebrae locations from input image data 401. The detection is accomplished by a feature matching between the input image data 401 and the vertebra templates 418. The features used are cross-modality features discussed in more detail below.

[0060] Global shape registration 404 aligns the detected landmarks from Landmark detection block 402 with the global spine model 420. Global shape registration 404 abstracts each vertebra as a 3D point so that the registration is completed by a point-based alignment which minimizes the landmark-vertebra point distances. In certain aspects, the registration is formulated as a coherence point drift process and is discussed in more detail below. The aligned spine model provides a coarse identification of vertebrae locations and labels. The 3D vertebrae models are then imposed on the model points.

[0061] Local pose adjustment 406 recovers the orientation and dimension of each vertebra. The adjustment determines the optimal alignment between the vertebrae and corresponding image structures, which is accomplished using triplanar model 422 and by a group-wise registration discussed in more detail below. The refined vertebrae are combined to construct a full 3D model of the spine 408. The spine model 408 contains intrinsic labels for each vertebra, thus the registration of the spine model 408 and vertebral landmarks immediately provides the vertebra detection and labeling.

[0062] Triplanar Model.

[0063] The triplanar model is designed for the joint representation of 3D shape and 3D appearance of a vertebra. In certain aspects, each triplanar vertebra model contains a 3D mesh model embedded with a triplanar template representation. The three planar templates in this model are three MR or CT patches describing the MR/CT appearance of a 3D vertebra projected on coronal, sagittal, and axial views, respectively. The detection of a vertebra landmark in an image is accomplished by searching of identical matching between the planar templates and the input image. In addition, the 2D warping of a planar template on the associated image plane is considered as applying a 3D deformation for the 3D vertebra model then projecting the appearance on that image plane.

[0064] 3D Spine Model.

[0065] The spine model used in global shape recognition 404 is built upon a 3D spine mesh model. The meshes are manually built from CT scans of a healthy spine. In certain aspects, each vertebra in the 3D spine mesh is individually built and manually assigned to construct the corresponding MR or CT triplanar vertebral model 422. In certain aspects, the meshes assigned and the corresponding triplanar templates 418 are aligned using 3D CAD software. The resulting meshes can be applied to both MR and CT matching as the obtained anatomic structure is independent to image modality. The 3D spine model 420 and the associated triplanar vertebra 422 are assigned with different deformation mechanisms and are unified in the Hierarchical Deformation Model 410.

[0066] Hierarchical Deformable Model

[0067] Hierarchical Deformable Model (HDM) 410 is used for the deformable matching of local/global spine structures. Combining triplanar vertebra models 422 under the organization of a 3D spine constructs a compositional structure that simulates the spine anatomical deformation mechanism for spine matching. The model contains three modules:

[0068] 1) Local appearance module 412 contains the planar templates 418 for each vertebra in each view. The templates are described by cross-modality features which unifies the appearances of both MR and CT. This module conducts the landmark detection using the planar templates 418.

[0069] 2) Local geometry module 414 contains the triplanar vertebra models 422 composed by the planar templates 418 from local appearance module 412. Local geometry module 414 conducts pose adjustment for each vertebra via the warping of planar templates 418. The warping is guided by the group-wise registration (joint-alignment) of the planar templates 418.

[0070] 3) Global Geometry Module 416 contains the set of connected triplanar vertebra models 422 organized under the spatial layout of a spine. This module conducts shape registration between detected landmarks from block 402 and the built-in vertebra models 420. The registration is implemented by a point-based registration process which aligns the abstracted vertebra points with the landmark points.

[0071] The advantages of using the hierarchical model includes: 1) weak supervision. HDM 410 borrows geometric deformation in handling vertebra detection so that training sophisticated vertebra detectors is no longer necessary. This solves the shortcomings of most learning-based methods, where a large number of samples and intense manual labeling is required in the training stages. 2) Versatile data adaptation. The triplanar representation can perform spine recognition on both specified MR/CT slices and whole image volume. User not only can obtain the recognition result significantly faster, but also can control the progressive spine shape approximation by feeding the model with more/less slices. 3) 3D reconstruction as a natural by-product. The spine detection and matching directly correspond to the deformations of a standard 3D spine model, which in turn, provides a model-based 3D reconstruction of the spine image.

[0072] Local Appearance Module for Vertebra Landmark Detection

[0073] Local appearance module 412 is used for extracting cross-modality features that robustly encode the vertebra appearances in different image modalities. The cross-modality features are obtained by fusing the image features from MR and CT using a multi-model deep network. Using the fused features, the vertebra landmark detection can be performed on both MR and CT data with improved accuracy.

[0074] Feature Extraction Using Multi-Modal Deep Networks

[0075] A multi-modal deep network model is applied for learning and extracting features across modalities. Cross-modality features are more reliable than single modality features in vertebra detection because some image structures in one modality will be nearly missing in another modality, i.e., vertebra discs features in MR scans will be missing in CT scans. Thus, single modality features from either MR or CT will be insufficient to describe the complete vertebra structure. Also, features from different modalities tend to compensate and enhance each other; for example, disc features from MR can help to identify the vertebra locations for CT scans as discs and vertebrae share a lot common image structures (i.e., vertebrae boundaries). Therefore, good vertebra features can be learned by fusing image features from all presenting modalities.

[0076] Network Design.

[0077] In certain aspects of the present disclosure, the multi-modal network design is different from previous work by building the network on two convolution restricted Boltzmann machine (CRBM) layers and two layers of restricted Boltzmann machine (RBM). Through the layer-wise iterative update of convolution filter-banks, the CRBMs adaptively extract significant 2D image features. For each image modality, a unique set of CRBMs is used for extracting translation-invariant features from that modality. Similar to CRBM, through a layer-wise update of their connecting weights, the RBMs provide a well-ordered representation of the input signals. Unlike single-modal training, in certain aspects of the present disclosure, the upper layer RBMs are trained by feeding them the extracted lower-layer MR and CT features together. The RBMs automatically mix the MR and CT features generating a unified representation for both MR and CT modalities. The complete learning process is conducted in an unsupervised fashion.

[0078] Feature Fusion.

[0079] Feature fusion is a unique property of deep network models. The purpose of feature fusion is to combine the common features shared among different modalities and to enhance the feature representation for capturing more image details. Traditional learning-based detection methods often rely on the direct classification of handcrafted features such as SIFT, HOG, or Haar functions. Because of the distinct appearances of original image modalities, the corresponding feature maps of the handcrafted features can be extremely diverse. The supervised classifiers will require a large amount of labeled data to resolve this diversity. In contrast to traditional methods, deep networks contain multi-layer abstraction and adaptive features tuning that automatically combine the higher layer feature representation of different modalities. These combinations also enhance the single modality features extracted from lower layers as the combined features can now represent additional image structures from other modalities, i.e., a better MR feature can be learned by using CT samples and vice versa. Therefore, the multi-modal deep network requires less training samples than the traditional approaches but still can obtain good discriminative features.

[0080] Configurations.

[0081] In certain aspects, the parameters of the deep network are as follows. Layer 1 CRBM consists of a 7×7×6 filter-bank and layer 2 CRBM consists of a 9×9×12 filter-bank. Each filtering is followed with a ½ sub-sampling and probability pooling. The resulting output of layer 2 is a 72-dimensional binary feature map with each map becoming ¼ of the original image. Layer 3 will map each 6×6×72 cube in layer 2 to a feature vector of 800 dimensions. The last layer (layer 4) will reduce the dimensions of the layer 3 vector to 200 dimensions. The final deep feature map is a 200-dimensional map where each map is of ¼ the original image size. i.e., every 24×24 image patch is encoded to a 200 descriptor vector through the network. In certain aspects, the CRBMs and RBMs are learned through a training set which contains 50 T1 MR slices, 50 T2 MR slices, and 100 CT slices. In certain aspects, all slices are sampled near the mid-plane in the image volume with the same size of 250×250, as pixel size is 1×1 mm. The MR features and CT features are separately learned then fused by the RBM layers.

[0082] Planar Template Construction

[0083] The planar appearance template used in landmark detection are the MR/CT template patches 418 in triplanar vertebra model 422. A planar template is generated by taking the mean of a set of training patches from different image slices, then map the mean patches to high dimensional feature map using the deep network. For example, the sagittal template of the lumbar vertebrae in CT is generated by taking the mean appearance of the set of lumbar image patches in the CT training set then convert it to feature maps. Note that in certain aspects, the training images are aligned beforehand by using the part-based registration model discussed below. In certain aspects, there are four different types of templates: lumbar (L), thoracic (T), cervical (C), and sacrum (S). Each template type is generated separately by the same approach. In certain aspects, there are a total of 4×2×3 templates that serve the four vertebra types (S, L, T, C), two image modalities (MR and CT), and three views (coronal, sagittal, and axial). Once the input image modality is identified from the image header, the associated templates are loaded in the triplanar model to perform the appearance matching.

[0084] Templates Vs Trained Classifiers.

[0085] The reason for choosing the template approach is that the invariant appearance of a vertebra pattern is unique to other local image appearances and most of the mismatching is caused by pose distortions. It is more efficient to discriminate vertebrae from local image structures using the invariant vertebra appearance along with pose deformations. Previous supervised learning-based detection methods focus on vertebra appearance only and resolve pose variations by training their classifiers with extra samples, especially large amount of pose varied samples. The use of planar template with unsupervised learned features and pose deformations not only reduces the workload of collecting large amounts of medical data with massive manual labelling, but also significantly simplifies the training process while still maintaining the performance of vertebrae detection.

[0086] Landmark Detection

[0087] The initial landmark detection is done by matching the vertebra appearance templates with the input image. The trained deep network is applied in this detection. According to the input image modality, layer 1 and 2 in the deep network is substituted with the related MR/CT CRBMs, and with RBM layers remained unchanged. Following the filtering and pooling process of the deep network, the input image slice is expanded to a set of high-dimensional feature maps.

[0088] Detection.

[0089] Using the deep-feature descriptor mentioned above, the template matching is done by comparing the L² distances of the feature vectors between input image and the appearance template. Suppose f_dp is the deep feature descriptor defined by the deep network, for a template T the matching response on point p in an input image I is:
r(p,T)=exp(−ε∥f_dp(I_p)−f_dp(T)∥²)  Equation 1:

[0090] where I_p is the image patch centered at p with the same size of T and c is a fixed constant.

[0091] The template T is deformed with a set of transforms to match different vertebrae with various poses. In certain aspects, can be a series of 2D rotations and re-scalings. After the basic matching for each input scan I, responses are synthesized for their corresponding image views and obtain the final 3D responses. The peak positions are the desired vertebra landmarks. Finer positioning is obtained by using a pose adjustment for each landmark as discussed below.

[0092] Local Geometry Module for Vertebra Pose Adjustment

[0093] The goal of the local geometry module is to estimate the 3D pose of each triplanar model for the best model-image alignment. In certain aspects, the pose of a triplanar vertebra model is defined as the orientation and the anisotropic scales of that model in 3D space. The 3D pose can be described by the projections of 3D vertebra on 2D planes, which is equivalent to the planar poses of the intrinsic templates from the triplanar model. The optimal 3D poses of the collected triplanar models are obtained by a group-wise registration of the planar templates, and then by the back-projections of the planar poses to 3D.

[0094] Planar Pose Representation

[0095] The planar pose of a triplanar vertebra model 422 is the 2D orientation and sizes of its three built-in planar templates. Due to the rigidity of vertebra model, the planar poses are defined as invertible 2D affine transforms. The invertibility of pose transforms implies that any arbitrary planar pose can be generated from one reference pose. In certain aspects, this assumption is used in estimating the planar poses of each triplanar vertebra model.

[0096] Generative Model.

[0097] For a given image view, e.g., sagittal view, the vertebra patterns that appear on the input slices are assumed to be generated by the same appearance template. The reference template is from the sagittal template patch of a triplanar model. According to repetition of vertebra patterns, we assume that each spine section (sacrum, lumbar, thoracic, and cervical) has its own reference appearance. Different vertebrae in the same section differ only by their poses. For example, a template of sagittal CT lumbar generates the repetitive lumbar vertebra patterns in sagittal CT slice with varied poses. Let Tε^d1×d2 be the planar template, d₁ and d₂ are the height and width of the template patch. The potential vertebra landmarks p on an arbitrary slice I satisfies:
T∘G_p=I_p  Equation 2:

[0098] where I_p is the local vertebra patch with arbitrary pose, and G_p is a geometric transform that warps T to I_p so that T ‘generates’ I_p via G_p. Stitching a set of deformed template replicas together can reconstruct the spine appearance.

[0099] Pose Geometry.

[0100] 2D transforms are defined on planar poses by an algebraic model. According to the generative model, the planar pose of a vertebra at p is uniquely determined by transform G_p. The G_p is defined as an affine transform which can be explicitly formulated as a 3×3 invertible transform matrix. All possible transform matrices form a Lie group Aff(2)⊂GL(3). Using the matric group form, G_pεAff(2) is formulated as:
G_p=Exp(Σ_k=1⁶a_p^kE_k)  Equation 3:

[0101] $\begin{array}{cc}\begin{array}{ccc}{E}_1=\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 0\end{array}\right)& E_2=\left(\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ 0& 0& 0\end{array}\right)& E_3=\left(\begin{array}{ccc}0& -1& 0\\ 1& 0& 0\\ 0& 0& 0\end{array}\right)\\ E_4=\left(\begin{array}{ccc}0& 1& 0\\ 1& 0& 0\\ 0& 0& 0\end{array}\right)& E_5=\left(\begin{array}{ccc}0& 0& 1\\ 0& 0& 0\\ 0& 0& 0\end{array}\right)& E_6=\left(\begin{array}{ccc}0& 0& 0\\ 0& 0& 1\\ 0& 0& 0\end{array}\right)\end{array}& \mathrm{Equation}4\end{array}$

[0102] where a_p=[a_p¹, . . . , a_p⁶] and the infinitesimal generators {E_k} form the Lie algebra aff(2) of Aff(2). The deformations on planar templates on different views will determine a 3D pose of the 3D vertebra model. This allows for the estimation the 3D pose through 2D methods as 2D models can be directly applied on single/multiple slice data or volume data.

[0103] The Lie group formulation is adopted by the model since the pose changes between adjacent vertebrae are smooth. Also, the planar deformations are considered to be smooth in local vertebra set and during the deformation process. This formula conforms to the kinematics model for robotics, where the continuous deformations on rigid structure are represented by smooth matrix groups. The pose of a planar template is parameterized by the 6 parameters in vector a_p. The pose of the associated 3D vertebra are controlled by 18 parameters, which correspond to the three degrees of freedom on sagittal, axial, and coronal views.

[0104] Planar Pose Adjustment by Group-Wise Registration

[0105] The recovery of template pose is accomplished by aligning the planar template to the identified landmark in the image. This can be understood as solving Equation 2 for landmark patch I_p when G_p is unknown. In addition, because of the repetitions of vertebra appearances, the alignment is not only applied on the landmark-template pairs but also between landmark pairs. The landmark-landmark alignment helps to enhance the regularity of landmark patches and reduce the ambiguity of the landmark identification. In certain aspects, a part-based group-wise registration is applied for regularizing the landmark detection using the vertebra repetition. The registration model is a variant of a congealing model. However, unlike the congealing model the proposed model is part-based and can be applied on repetitive image patches.

[0106] Mathematical Formulation.

[0107] Given a set of sampled scans from the same image view, an initial landmark set is identified such that I_p is a deformable landmark patch observed on pε, p is represented as the planar coordinates of the image view. Suppose T is the corresponding template for the identified landmarks, then the alignment is formulated as the minimization of the function:
u(G_p)=φ_T(G_p)+Σ_(p,q)εψ(G_p,G_q)  Equation 5:

[0108] where φ is a data term and ψ is a smoothness term for neighbor set . Let I_p∘G_p⁻¹ and T be vectors in ^d1×d2, φ and ψ are then defined as:
φ_T(G_P)=∥I_p∘G_p⁻¹−T∥²  Equation 6:
ψ(G_p,G_q)=∥I_p∘G_p⁻¹−I_q∘G_q⁻¹∥².  Equation 7:

[0109] The expression of I_p∘G_p⁻¹ means the deformed landmark patch I_p is warped by transform G_p⁻¹ to exactly the same size of T. In other words, I_p∘G_p⁻¹ and I_q∘G_q⁻¹ are of the same size such that direct subtraction is possible. To make the registration more robust, certain aspects encode the patch by the deep feature descriptor described above. Each landmark patches become a feature patch written as f_dp(I_p∘G_p⁻¹). Note that G_p is parameterized by a_p, and is written: F_p(a_p)=f_dp(I_p∘G_p⁻¹). Therefore, for a λ_u>0, Equation 5 becomes a functional variable a_p as
u(a_p)=φ_T(a_p)+Σ_(p,q)εψ(a_p,a_q)  Equation 8:
φ_T(a_p)=∥F_p(a_p)∘G_p⁻¹−f_dp(T)∥²  Equation 9:
ψ(a_p,a_q)=∥F_p(a_p)−F_q(a_q)∥².  Equation 10:

[0110] In certain aspects, the functional u is solved by Gauss-Newton method, with the alignment being taken over each p in sampled image scans. The pose change Δa_p is obtained via
Δa_p=(^T−1Td(a)  Equation 11:
d(a)=F_p(a_p)−f_dp(T)+Σ_(q,p)ε(F_p(a_p)−F_q)  Equation 12:

[0111] $\begin{array}{cc}J\left(a_p\right)={\left(\frac{\partial d\left(a_p\right)}{\partial a_p^1},\dots ,\frac{\partial d\left(a_p\right)}{\partial a_p^k}\right)}^T& \mathrm{Equation}13\end{array}$

[0112] The coefficient vector a_p is iteratively updated: a_p+Δa_p, leading to the progressive alignment of the vertebra parts. The updated a_p is substituted back into equation 3, warping the planar temple and the 3D vertebra. In certain aspects, the part-based registration can be applied on landmark patches within a single image scan or those distributed among multiple scans. This allows for the conducting the registration on arbitrary image views as some repetitions do not appear in the single image.

[0113] Estimation of 3D Pose

[0114] Through the group-wise registration on different image views, the planar pose of each potential landmark patches are described by the optimal vector a_p obtained by equation 8. The planar poses are then back-projected from three views (coronal, sagittal, axial) to synthesize a 3D pose so that the dimensions and orientation of a triplanar vertebra model are fully recovered. The estimation problem is similar to that found in visual serving, where the 3D transform is recovered by projection 2D planar deformations.

[0115] From Aff(2) to SE(3).

[0116] The 3D vertebra model is a rigid structure whose pose is defined by an invertible 3D rotation and a translation. This is a different assumption from that if the planar template assumption as the planar template is allowed to both rotate and dilate under affine transforms. For a potential landmark, which is identified in three image views, let p denote the landmark center. The possible 3D transforms form a Lie group SE(3) such that R_pεSE(3) is a 4×4 transform with a matric group representation similar to equation 3:
R_p=Exp(Σ_k=1⁶c_p^kV_k)  Equation 14:

[0117] $\begin{array}{cc}\begin{array}{cc}\begin{array}{ccc}{V}_1=\left(\begin{array}{cccc}0& -1& 0& 0\\ 1& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right)& V_2=\left(\begin{array}{cccc}0& 0& 1& 0\\ 0& 0& 0& 0\\ -1& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right)& V_3=\left(\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& -1& 0\\ 0& 1& 0& 0\\ 0& 0& 0& 0\end{array}\right)\\ V_4=\left(\begin{array}{cccc}0& 0& 0& 1\\ 0& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right)& V_5=\left(\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& 0& 1\\ 0& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right)& V_6=\left(\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 1\\ 0& 0& 0& 0\end{array}\right)\end{array}& \end{array}& \mathrm{Equation}15\end{array}$

[0118] For a triplanar vertebra model, the three template patches inside the model is rotated and translated according to the pose of the 3D vertebra, i.e., the warped planar landmark obtained from group-wise registration is the projection of the corresponding template patch in the vertebra model. The corners of the built-in three template patches define a 3D bounding box for the 3D vertebra model. The template-landmark projection indicates that the projections of the 3D corners also define the warped bounding boxes of the planar landmarks. Utilizing the 2D-3D projection between corners, the 3D pose of the vertebra model is recovered.

[0119] Estimation Formulation.

[0120] Let p_s be the 3D corner of the sagittal template patch in the triplanar vertebra model. The orthographic projection P_s such that P_sp_s, is the 2D corner of the warped landmark bounding box on the projected sagittal image plane. The projection of sagittal corners, axial corners, and coronal corners respectively as {circumflex over (p)}_s, {circumflex over (p)}_a, {circumflex over (p)}_c
{circumflex over (p)}_s=P_sp_s
{circumflex over (p)}_a=P_ap_a
{circumflex over (p)}_c=P_cp_c  Equation 16:

[0121] Note that any 3D corner p and 2D corner {circumflex over (p)} in the above equations are represented in homogeneous coordinates, and projection P is a 4×4 matrix which agrees with the matrix definition in equation 14. In certain aspects, the initial positions of the 3D corners are P_s0, P_a0, p_c0, the relations of the template-landmark projection is described as

[0122] $\begin{array}{cc}e\left(c\right)=\sum _{i=1}^4{P_{S}R{p}_{s0}^i-{\hat{p}}_s^i}^2+{P_{a}R{p}_{a0}^i-{\hat{p}}_a^i}^2+{P_{c}R{p}_{c0}^i-{\hat{p}}_c^i}^2& \mathrm{Equation}17\end{array}$

[0123] where p¹, . . . , p⁴ represent the four corners of the template patch. * is the desired 3D pose such that
*=arg min_εSE(3){e()}  Equation 18:

[0124] Similar to equation 8, in certain aspects, equation 17 is represented in Lie algebra form to enhance the smoothness of transformation. Substituting equation 14 into equation 17 yields:

[0125] $\begin{array}{cc}e\left(c\right)=\stackrel{4}{\sum _{i=1}}(\begin{array}{c}{P_s\left(\mathrm{Exp}\left(\sum _{k=1}^6{c}^k{V}_k\right)\right)p_{s0}^i-{\hat{p}}_s^i}^2+\\ {P_a\left(\mathrm{Exp}\left(\sum _{k=1}^6{c}^k{V}_k\right)\right)p_{a0}^i-{\hat{p}}_a^i}^2+\\ {P_c\left(\mathrm{Exp}\left(\sum _{k=1}^6{c}^k{V}_k\right)\right)p_{co}^i-{\hat{p}}_c^i}^2\end{array})& \mathrm{Equation}19\end{array}$

[0126] The optimal c* arg min {e(c)} is obtained by solving equation 19 using the Gauss-Newton method.

[0127] Global Geometry Module for Spine Shape Registration

[0128] Global geometry module 414 performs the global shape registration for the detected vertebra landmarks. This module is built upon the 3D spine model and its shape recognition. The 3D spine model is formed by the combined local triplanar vertebra models and is abstracted as a pint-connected global model. Shape registration of the spine model provides a top-down point matching for the potential landmarks. The global module is described by three parts: the unification of triplanar models that forms the global spine, the vertebra parsing that removes outlier landmarks, and the non-rigid deformation of global spine model that conducts the shape registration.

[0129] Unification of Local Models

[0130] Similar to the generative model of the local triplanar model, the global spine model is described as a higher order generative model. In certain aspects, a graphical probability model is applied to formulate the higher order generative model. Similar to the compositional anatomic structure of the spine, the 3D model (without point abstraction) in a HDM is composed by the set of 3D triplanar vertebra models. The ith vertebra in the spine model is the local triplanar geometry model whose pose is C_i and has triplanar appearance Γ_i: C_i=(z_i,c_i),Γ_i={T_i^v} where z_i is the vertebra center location in ³, c_i represents the 3D pose as in equation 19, and set {T_i^v} contains the triplanar templates for vε{sagittal,axial,coronal}. The whole spine global model which contains m vertebrae can be represented as a vertebra set {C₁, . . . , C_m}. In certain aspects, landmark detection obtains a set of potential landmarks ⊂. A detected landmark at xε whose configuration is described by its pose and appearance C_x=(x, c_x), Π_x={I_x^v} where {I_x^v} are the deformed planar patches around x from different image views v.

[0131] In certain aspects, each detected landmark x can only correspond to one triplanar model with one pose. In model-landmark matching, an observed local landmark and the planar image patches around the landmark are considered as being generated from one of the triplanar vertebrae models in the global spine model. A probability density function about landmark x can be constructed:
p(x)=p(Π_x,C_x;{Γ_i,C_i})  Equation 20:

[0132] where {C_i}_{i=1, . . . m} is the set of vertebrae in the global model. The formulation implies that the density function will increase when local landmark at xε is highly correlated to the global model. The correlation can be understood as a deformable matching when deformable model is driven by the force of likelihood. The driving force of x is tuned specifically for a single triplanar model in the global model. Equation 20 can be explicitly defined in prior-likelihood form:
p(Π_x,C_X;{Γ₁,C_i})=p(Π_x,C_x|{Γ_i,C_i})p({Γ_i,C_i})  Equation 21:

[0133] where p({Γ_i,C_i}) represents the prior initialization of the global model. In certain aspects, a latent variable is introduced: L: →{1, . . . , m} to represent the membership index of vertebra models ({Γ₁,C_i}) from the global spine, such that the landmark on x is considered as generated from the L(x)-th vertebra model. In addition, L satisfies
L(z_i+1)=L(z)+1,i=1, . . . ,m−1
L(z_i−1)=L(z)−1,i=1, . . . ,m  Equation 22:

[0134] where {z_i} belongs to the global model {C₁, . . . , C_m}. Given an initialized global model, the planar appearances and pose of the local landmark at x are generated by a mixture model
p(Π_x,C_x|{Γ_i,C_i})=Σ_L=1^mp(L)p(Π_x,L|{Γ_i,C_i})p(C_x|L,{C_i})   Equation 23:

[0135] where p(L) is the latent variable describing the association of current landmark and the specific vertebra model.

[0136] In the formulation of equation 23, p(L) can be defined as uniform distribution 1/m and the deformable matching will reduce to blind registration. Instead of uniform distribution, in certain aspects, p(L) can be defined as the confidence of local spatial compatibility where the explicit definition is defined below. The confidence is adaptive according to vertebra model type, so that some special landmarks can have stronger attraction to the corresponding vertebra models. The landmark-model correspondence of planar appearances and the correspondence of poses are defined respectively as

[0137] $\begin{array}{cc}p\left(\prod _x|L,\left\{{\Gamma }_i^v,C_i\right\}\right)=\prod _{v}p\left(I_x^v|L,\left\{T_i^v,a_i^v\right\}\right)=\prod _{v}p\left(I_x^v|T_i^v,a_i^v\right)=\frac{1}{z_{\Pi }}\prod _v\exp \left(-\mathrm{\alpha \varphi }{T}_L^v\left(a_L^v\right)\right)& \mathrm{Equation}24\end{array}$

[0138] and

[0139] $\begin{array}{cc}\begin{array}{c}p\left(C_x|L,\left\{C_i\right\}\right)=p(xL,\left\{z_i,c_i\right\}p\left(c_x|L,\left\{c_i\right\}\right)\\ =p(xL,\left\{z_i\right\}\prod _{v}p\left(a_x^v|L,\left\{a_i^v\right\}\right)\\ =p\left(x|z_L\prod _v\prod _{j\in \left(i\right)\bigcup i}p\left(a_x^v|a_i^v\right)\right)\\ =p(x|z_L\times \frac{1}{z_a}\prod _v\\ \exp \left(-\gamma \sum _{j\in \left(i\right)\bigcup i}\psi \left(a_x^v,a_j^v\right)\right))\end{array}& \mathrm{Equation}25\end{array}$

[0140] where φ and ψ are defined in equation 8. Thus, the local geometry alignment (equation 8) is integrated into the global model of equation 21, and the local alignment now contributes to the global deformable matching.

[0141] Since the global spine model is represented as the set of connected triplanar vertebra models, the deformation of the global model corresponds to the collaborated deformations of triplanar models which is highly complicated. The graphical probability model can not only unify the appearance-pose representations among local models, but can also unify the local and global deformation formulations in a single formulation. The interactions between global spine and local vertebra can be easily accomplished using this formulation.

[0142] Vertebra Parsing

[0143] Vertebra parsing is the verification for removing falsely detected landmarks using the spatial inter-vertebra correspondences. It reduces the ambiguity of spine shape registration. There are two spatial clues that help to verify the correct landmarks: 1) Pose compatibility of adjacent landmark. As the spine is a part-connected structure, the correct match of a landmark and a vertebra model suggests that the spatial organization of nearby landmarks should also be compatible with the organization of the adjacent vertebra models in the global spine model. Outliers can be easily removed by its spatial compatibility. 2) Anchor vertebra. The anchor vertebrae are the vertebrae whose appearances are significantly different from the other vertebra in the spine. These special vertebra are used to identify the spine section (lumbar or cervical) of the input data. In certain aspects, the anchor vertebrae used are (a) the S1 for identifying lumbar related sections; (b) the C1+C2 (combined as one model) for identifying cervical related sections.

[0144] The overall vertebra parsing is carried out as: Identify spine section by anchor vertebra and remove outliers that are not compatible with local spatial organization. Vertebra parsing is formulated in the form of graphical model of equation 20. Let p(L) represent the classification of landmark xε as vertebra L(x)ε{1, . . . m}. L is considered a Markov random field (MRF) over
p(L)=Π_xεp(L(x)|L(y))  Equation 26:

[0145] where {L(y)}_yεk(x) represents the labels assigned to a modified neighbor set _k(x). Unlike the traditional definition of neighbor set (x), _k(x) is defined as
_k(x)=(x+kR(x)d),kε{−K, . . . ,K}  Equation 27:

[0146] where k>0 is an integer and dε² is a unit vector. _k is the neighbors of x obtained by shifting x with displacement d oriented by R(x), where R(x) is the planar pose obtained from equation 18. The potential landmark location is confirmed only if its _±1 neighbors are in compatible locations.

[0147] The probability of p(L(x)|L(y)) is defined as:

[0148] $\begin{array}{cc}p\left(L\left(x\right)|L\left(y\right)\right)=\frac{1}{z_L}\exp \left(p\sum _{k=-K}^K\sum _{y\in k_{}\left(x\right)}\delta \left(L\left(x\right)+k,L\left(y\right)\right)\right)& \mathrm{Equation}28\end{array}$

[0149] where δ is the Kronecker delta function. The definition indicates the x can be labeled as the L(x)-th vertebra only if its shifted position y is a potential landmark labeled as L(x)+k. The parsing of L(x) is then done by maximizing the log-likelihood of equation 28, which, in certain aspects, can be implemented by a linear searching algorithm. The outliers are eliminated after parsing, leaving only the correct vertebra landmarks.

[0150] Registration for Spine Shape

[0151] The shape registration of spine model and detected landmarks are implemented by point-based registration. In this registration, each vertebra models in the spine model are abstracted as points to match the identified landmarks obtained from vertebra parsing. The registration is driven by minimizing point pair inter-distances modulated with landmark alignment and appearance matching. The deformation applied in our registration is inspired by coherence point drift (CPD).

[0152] As the registration is point-based, the explicit point correlation needs to be defined. Following the formulation of generative mixture mode in Equation 23, the explicit definition of p(x|z_L) in Equation 25 becomes:

[0153] $\begin{array}{cc}p\left(x|z_L\right)=\frac{1}{{\left(2{\mathrm{\pi \sigma }}^2\right)}^{\frac{3}{2}}}\exp \left(-\frac{x-{(z_L+v\left(z_L\right)}^2}{2{\sigma }^2}\right)& \mathrm{Equation}29\end{array}$

[0154] where v(z_L) is the displacement of the original vertebra location z_L. The likelihood above represents the attraction between x and z_L, which in certain aspects, can be understood as a Gaussian Mixture Model (GMM). A variation for the position z_L+v(z_L) or for the standard deviation σ will cause the likelihood to change accordingly, which implies the mode of z_L towards or away from x. The goal is then to search for the optimal v and σ. The objective function can be written as:

[0155] $\begin{array}{cc}\begin{array}{c}Q\left(v,\sigma \right)=-\log \prod _{x}p\left(\prod _x,C_x|\left\{{\Gamma }_i,C_i\right\}\right)\\ =\sum _x\log \sum _{L}p\left(L\right)p\left(\prod _x|L,\left\{{\Gamma }_i,C_i\right\}\right)p\left(C_x|\left\{C_i\right\}\right)\\ =-\sum _x\log \sum _L\theta \left(x,L\right)p\left(x|z_L\right)\end{array}& \mathrm{Equation}30\end{array}$

[0156] where
Θ(x,L)=p(L)p(Π_x|L,{Γ_i,C_i})Π_vΠ_jε(i)∪ip(a_x^v,a_j^v)  Equation 31:

[0157] and p(L), p(Π_x|L,{Γ_i,C_i}), p(a_x^v,a_j^v) an are defined in Equation 28, 24, and 25, respectively. In certain aspects, the objective function in Equation 30 can be minimized using the expectation-maximization (EM) algorithm, which iteratively updates the parameters in Equation 30 to solve v and σ. Using the EM algorithm formulation, Equation 30 can be reformulated as an equivalent minimization objective function:
Q=−Σ_xΣ_Lp^old(z_L|x)log(Θ^new(x,L)p^new(x,z_L))  Equation 32:

[0158] where Θ^new(•,•) and p^new(•,•) represent the prior and likelihood in the M-step using the newly evaluated parameters. p^old(z_L|x) is the posterior of the E-step which is evaluated using the old parameters. By minimizing Equation 32 instead of Equation 30 the same optimal v and σ can be obtained. From Equation 29 and Equation 25, Equation 32 can be revised as:

[0159] $\begin{array}{cc}Q\left(v,\sigma \right)=\sum _x\sum _L{p}^{\mathrm{old}}\left(z_L|x\right)\left(\frac{{x-z_L-v\left(z_L\right)}^2}{2{\sigma }^2}-\log \left(\Theta \left(x,L\right)\right)\right)+\frac{3N_p}{2}\log {\sigma }^2& \mathrm{Equation}33\end{array}$

[0160] $p^{\mathrm{old}}\left(z_L|x\right)=\frac{\exp \left(-\frac{{x-z_L-v^{\mathrm{old}}\left(z_L\right)}^2}{2{\left({\sigma }^{\mathrm{old}}\right)}^2}\right)\Theta \left(x,L\right)}{\exp \left(-\frac{{y-z_L-v^{\mathrm{old}}\left(z_L\right)}^2}{2{\left({\sigma }^{\mathrm{old}}\right)}^2}\right)\Theta \left(y,L\right)}$

[0161] where N_p=Σ_xΣ_Lp^old(z_L|x).

[0162] In certain aspects, the variance v is assumed to be smooth, so that different point x will not collapse and their velocity flow will not interfere with each other in deformation. In certain aspects, a coherence regularization term is added:

[0163] Equation 35:

[0164] $Q\left(v,\sigma \right)=\sum _x\sum _L{p}^{\mathrm{old}}\left(z_L|x\right)\left(\frac{{x-z_L-v\left(z_L\right)}^2}{2{\sigma }^2}-\log \left(\Theta \left(x,L\right)\right)\right)+\frac{3N_p}{2}\log {\sigma }^2+\frac{\lambda }{2}\sum _x{v\left(z_L\right)}_H^2$

[0165] where λ>0. ∥•∥ is the norm for a Hilbert space .

[0166] To obtain the optimal v, σ is fixed in Equation 35 then from the Euler-Lagrange equation:

[0167] $\begin{array}{cc}\frac{1}{2{\sigma }^2\lambda }\sum _x\sum _L{p}^{\mathrm{old}}\left(z_L|x\right)\left(x-z_L-v\left(z_L\right)\right)\delta \left(y-z_L\right)=\sum _{k=0}^{\infty }{\left(-1\right)}^k{\beta }_k{D}^{2k}v\left(y\right).& \mathrm{Equation}36\end{array}$

[0168] The solution is derived from the Green's function of the differential operator on the right side:

[0169] $\begin{array}{cc}\frac{1}{2{\sigma }^2\lambda }\sum _x\sum _L{p}^{\mathrm{old}}\left(z_L|x\right)\left(x-z_L-v\left(z_L\right)\right)K\left(y,-z_L\right)& \mathrm{Equation}37\end{array}$

[0170] where K is the Green's function. σ can be obtained by choosing a Gaussian function for K and fix v:

[0171] $\begin{array}{cc}{\sigma }^2=\frac{1}{3N_p}\sum _{x\in X}\sum _L{x-z_L-v\left(z_L\right)}^2& \mathrm{Equation}38\end{array}$

[0172] The updated v and σ are substituted in Equation 34 and then continue to compute the new v and σ in the next iteration.

[0173] In these examples, the machine readable instructions comprise a program for execution by a processor such as processor 512 shown in the example processor platform 500 discussed below in connection with FIG. 5. The program can be embodied in software stored on a tangible computer readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), a BLU-RAY™ disk, or a memory associated with processor 512, but the entire program and/or parts thereof could alternatively be executed by a device other than processor 512 and/or embodied in firmware or dedicated hardware. Further, although the example program is described with reference to the flowchart illustrated in FIG. 4, many other methods of implementing the example multi-kernel learning can alternatively be used. For example, the order of execution of the blocks can be changed, and/or some of the blocks described can be changed, eliminated, or combined.

[0174] As mentioned above, the example processes of FIG. 4 can be implemented using coded instructions (e.g., computer and/or machine readable instructions) stored on a tangible computer readable storage medium such as a hard disk drive, a flash memory, a read-only memory (ROM), a compact disk (CD), a digital versatile disk (DVD), a cache, a random-access memory (RAM) and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the term tangible computer readable storage medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media. As used herein, “tangible computer readable storage medium” and “tangible machine readable storage medium” are used interchangeably. Additionally or alternatively, the example processes of FIG. 4 can be implemented using coded instructions (e.g., computer and/or machine readable instructions) stored on a non-transitory computer and/or machine readable medium such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random-access memory and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media. As used herein, when the phrase “at least” is used as the transition term in a preamble of a claim, it is open-ended in the same manner as the term “comprising” is open ended.

[0175] The subject matter of this description may be implemented as stand-alone system or for execution as an application capable of execution by one or more computing devices. The application (e.g., webpage, downloadable applet or other mobile executable) can generate the various displays or graphic/visual representations described herein as graphic user interfaces (GUIs) or other visual illustrations, which may be generated as webpages or the like, in a manner to facilitate interfacing (receiving input/instructions, generating graphic illustrations) with users via the computing device(s).

[0176] Memory and processor as referred to herein can be stand-alone or integrally constructed as part of various programmable devices, including for example a desktop computer or laptop computer hard-drive, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), programmable logic devices (PLDs), etc. or the like or as part of a Computing Device, and any combination thereof operable to execute the instructions associated with implementing the method of the subject matter described herein.

[0177] Computing device as referenced herein may include: a mobile telephone; a computer such as a desktop or laptop type; a Personal Digital Assistant (PDA) or mobile phone; a notebook, tablet or other mobile computing device; or the like and any combination thereof.

[0178] Computer readable storage medium or computer program product as referenced herein is tangible (and alternatively as non-transitory, defined above) and may include volatile and non-volatile, removable and non-removable media for storage of electronic-formatted information such as computer readable program instructions or modules of instructions, data, etc. that may be stand-alone or as part of a computing device. Examples of computer readable storage medium or computer program products may include, but are not limited to, RAM, ROM, EEPROM, Flash memory, CD-ROM, DVD-ROM or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired electronic format of information and which can be accessed by the processor or at least a portion of the computing device.

[0179] The terms module and component as referenced herein generally represent program code or instructions that causes specified tasks when executed on a processor. The program code can be stored in one or more computer readable mediums.

[0180] Network as referenced herein may include, but is not limited to, a wide area network (WAN); a local area network (LAN); the Internet; wired or wireless (e.g., optical, Bluetooth, radio frequency (RF)) network; a cloud-based computing infrastructure of computers, routers, servers, gateways, etc.; or any combination thereof associated therewith that allows the system or portion thereof to communicate with one or more computing devices.

[0181] The term user and/or the plural form of this term is used to generally refer to those persons capable of accessing, using, or benefiting from the present disclosure.

[0182] FIG. 5 is a block diagram of an example processor platform 500 capable of executing the instructions of FIG. 4 to implement the example multi-modality vertebrae recognition engine of FIG. 3. The processor platform 500 can be, for example, a server, a personal computer, a mobile device (e.g., a cell phone, a smart phone, a tablet such as an IPAD™), a personal digital assistant (PDA), an Internet appliance, or any other type of computing device.

[0183] The processor platform 500 of the illustrated example includes a processor 512. Processor 512 of the illustrated example is hardware. For example, processor 512 can be implemented by one or more integrated circuits, logic circuits, microprocessors or controllers from any desired family or manufacturer.

[0184] Processor 512 of the illustrated example includes a local memory 513 (e.g., a cache). Processor 512 of the illustrated example is in communication with a main memory including a volatile memory 514 and a non-volatile memory 516 via a bus 518. Volatile memory 514 can be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM) and/or any other type of random access memory device. The non-volatile memory 516 can be implemented by flash memory and/or any other desired type of memory device. Access to main memory 514, 516 is controlled by a memory controller.

[0185] Processor platform 500 of the illustrated example also includes an interface circuit 520. Interface circuit 520 can be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), and/or a PCI express interface.

[0186] In the illustrated example, one or more input devices 522 are connected to the interface circuit 520. Input device(s) 522 permit(s) a user to enter data and commands into processor 512. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint and/or a voice recognition system.

[0187] One or more output devices 524 are also connected to interface circuit 520 of the illustrated example. Output devices 524 can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display, a cathode ray tube display (CRT), a touchscreen, a tactile output device, a light emitting diode (LED), a printer and/or speakers). Interface circuit 520 of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip or a graphics driver processor.

[0188] Interface circuit 520 of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem and/or network interface card to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network 526 (e.g., an Ethernet connection, a digital subscriber line (DSL), a telephone line, coaxial cable, a cellular telephone system, etc.).

[0189] Processor platform 500 of the illustrated example also includes one or more mass storage devices 528 for storing software and/or data. Examples of such mass storage devices 528 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, RAID systems, and digital versatile disk (DVD) drives.

[0190] Coded instructions 532 of FIG. 4 can be stored in mass storage device 528, in volatile memory 514, in the non-volatile memory 516, and/or on a removable tangible computer readable storage medium such as a CD or DVD.

[0191] Thus, certain examples provide a clinical knowledge platform that enables healthcare institutions to improve performance, reduce cost, touch more people, and deliver better quality globally. In certain examples, the clinical knowledge platform enables healthcare delivery organizations to improve performance against their quality targets, resulting in better patient care at a low, appropriate cost.

[0192] Certain examples facilitate improved control over process. For example, certain example systems and methods provide the ability to perform automatic vertebra recognition to identify the global spine and local vertebra structural information such as spine shape, vertebra location and pose. Automatic spine recognition which supports quantitative measurement is essential in numerous spine related applications in orthopedics, neurology, and oncology. The task of automatic spine recognition is to extract the set of numerical parameters that can uniquely determine the global structure of the spine and certain local structures of the vertebrae. Currently, spine recognition is often simplified as vertebra detection, which extracts the locations and labels of the vertebrae in input images.

[0193] Certain examples leverage information technology infrastructure to standardize and centralize data across an organization. In certain examples, this includes accessing multiple systems from a single location, while allowing greater data consistency across the systems and users.

[0194] Technical effects of the subject matter described above may include, but is not limited to, providing systems and methods of automatic vertebra recognition to identify the global spine and local vertebra structural information such as spine shape, vertebra location and pose.

[0195] Moreover, the system and method of this subject matter described herein can be configured to provide an ability to better understand large volumes of data generated by devices across diverse locations, in a manner that allows such data to be more easily exchanged, sorted, analyzed, acted upon, and learned from to achieve more strategic decision-making, more value from technology spend, improved quality and compliance in delivery of services, better customer or business outcomes, and optimization of operational efficiencies in productivity, maintenance and management of assets (e.g., devices and personnel) within complex workflow environments that may involve resource constraints across diverse locations.

[0196] This written description uses examples to disclose the subject matter, and to enable one skilled in the art to make and use the invention. The patentable scope of the subject matter is defined by the following claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.