Apparatus for determining a three dimensional shape of an object

[0001] This application claims the benefit of U.S. Provisional Application No. 60/611,364, filed on Sep. 18, 2004 which is expressly incorporated herein.

[0002] The present invention relates generally to the capturing of three dimensional data of the shape of an object, and more particularly involves the use of cameras (or imagers) and light projectors to capture images of an object and storing that data for processing into a three-dimensional model of the shape of the object. In a preferred embodiment the invention is useful in fitting amputees with properly fitted prosthetic devices.

[0003] It is known to sense the shape of body parts or limbs three-dimensionally by means of active laser scanners. See, for example, PCT Publication WO92/08175. It is also known to use photogrammetric systems employing passive stereo cameras and thin, tight fitting envelopes of material carrying a high contrast pattern worn over the body part being sensed. See, for example, U.S. Pat. No. 5,911,126.

[0004] The present invention employs light projectors preferably mounted on or in a structure which is preferably in the shape of a ring, and cameras preferably mounted on or in the ring. The structure is placed over the object to be sensed and light planes are projected onto the object. The cameras and projectors are preferably electronically controlled to operate in a predetermined sequence to capture images of the light planes projected onto the object.

[0005] The structure is preferably connected to a computer and display. A processor analyzes the data captured to create a three-dimensional model of the object. To create the 3D model the invention preferably uses data taken from the two-dimensional edges of the light planes on the object from the cameras positioned at various angles to the object and converts the data to a 3D model.

[0006] In one preferred embodiment the 3D data model may be used in the manufacture of properly fitted prosthetic devices for an amputee. A prosthetist may operate the mobile apparatus of the present invention wherever the amputee may happen to be located. Data collected from the use of the apparatus of the present invention may be stored in a computer used by the prosthetist. Later the prosthetist may download the data stored on an amputee to be used in the manufacture of a properly fitted prosthetic device for the amputee.

[0007] Properly fitting an amputee with a prosthetic device is important to the comfort and health of the amputee as well as to the usefulness of the prosthetic device. An improperly fitted prosthetic device can cause discomfort to the amputee, and can cause sores or blisters to develop on or near the amputee's stump. By mapping the various contours of an amputee's stump in a three-dimensional data model that can later be downloaded, a manufacturer of prosthetic devices can produce a better fitting prosthetic device for the amputee that recognizes and accommodates substantially the same contours of the amputee's stump.

[0008] The present invention may be used directly on an amputee's stump or in association with a covering or liner worn over the stump. There are very few limits on the use of the present invention. It is really only limited in its use by the size of the structure and the object being sensed. The object needs to be able to fit in the structure about which the cameras and projectors are arranged.

[0021] With reference to the figures, there is shown in FIG. 1 a preferred embodiment of the apparatus 10 of the present invention. It comprises a ring structure 20 having a plurality of cameras 12 and projectors 14. The ring structure 20 is placed around an object 16 to be sensed or measured. Light planes 18 from the projectors 14 are aimed at the object 16 and the cameras 12 are actuated to gather and store digital images of the object 16 having the light planes projected thereon. Since the cameras 12 are taking images from several angles around the object 16 data is collected from practically all directions around the object. The data collected by the cameras 12 may be stored in a database associated with the ring structure itself or connected to the ring structure. A personal computer (PC) 30 may be electronically connected 22 to the ring structure 20 to analyze the data and display 40 a 3D image of the object. The object may be practically any object able to be placed inside the dimensions of the ring.

[0022] In one embodiment of the present invention the object 16 is a stump of an amputee. A prosthetist may operate the ring structure to gather data to form a 3D model of the amputee's stump. The data collected may be used in the manufacture of a prosthetic device for the amputee. The stump may be covered in advance of the image gathering by a gel liner common to the industry of prosthetics, or by other covering means.

[0023] The present invention is preferably comprised of cameras, projectors, and a ring structure. There are preferably four cameras built into the structure. On a target object placed at the center of the ring, each camera is preferably able to ‘see’ approximately 10″ up, 10″ down, and 7″ from side to side. The cameras should be focused for the proper exposure and synchronized for operation. There are preferably four projectors built into the structure. Each projector is preferably able to project 21 planes of light in towards the center of the ring. The planes of light are preferably evenly spaced and preferably fan out in an evenly spaced manner. For a model approximately 4″ in diameter, the projected planes of light will preferably illuminate a 20″ range (10″ up and 10″ down) of the model. Due to the high amount of current necessary to flash a projector, a large capacitor is preferably mounted on the projector card and fully charged before initiating a flash. The inside (working) surface of the ring is preferably approximately 25″ in diameter. The ring preferably houses, as noted above, four cameras and four projectors evenly spaced in an alternating sequence around the ring. Each camera and projector is preferably connected, in a daisy-chain arrangement, to a USB/processor board 24 inside the structure, which ultimately controls the cameras and projectors based upon commands sent over a USB channel from the user's PC.

[0024] The projector and camera arrangement, and usage are preferably as follows: The four projectors may be arranged in global orientation and designated North, South, East, and West. The four cameras may be arranged between the projectors and designated Northeast, Southeast, Northwest, and Southwest. While holding the structure normally, preferably using handgrips on the structure, the South projector is close to the operator's stomach, the East projector is close to the operator's right hand, the West is close to the operator's left hand, and the North is on the opposite side of the ring. The Northeast camera is between the North and East projectors, and so on.

[0025] Image capture is accomplished using the cameras. The cameras are preferably capable of taking a single full-frame (640×480) monochrome capture and sending it to the operator's PC. A capture can be issued for an individual camera or for all cameras simultaneously. The image data may be encrypted on the camera card itself and decoded on the user's PC.

[0026] For 3D capture and projector calibration purposes, the cameras are preferably capable of taking a ‘triple capture’—three consecutive frames of video coordinated with the projectors in the following manner: the North and South projectors will be on during the first frame, no projectors will be on during the second frame, and the East and West projectors will be on during the third frame. Due to potential memory limitations on the camera cards, the cameras preferably only store every third row of pixels for each frame. As with the full-frame capture, a triple-capture may be issued for an individual camera or for all cameras simultaneously and the data may be encrypted for protection during data transfer.

[0027] Once a triple capture is acquired, the software may process it in many ways. FIGS. 8A, 8B and 8C are a sampling of processed frames using a single triple capture from the northeast camera. The first three frames represent the raw data compressed into the triple capture.

[0028] Post-Processed—The ‘East Delta’ frame or FIG. 9 subtracts the ambient frame from the east-lit frame. Notice that all of the background objects and even the lights in the room have been nullified. Also notice the bright light to the side of the model—this is the west projector firing.

[0029] Post-Processed—The ‘North Delta’ frame or FIG. 10 subtracts the ambient frame from the north-lit frame. The bright light to the side of the model (object) is the south projector firing.

[0030] Post-Processed—The ‘Both’ frame or FIG. 11 is the addition of the two delta frames. It is used in diagnostics to confirm that the projected stripes are covering the model properly.

[0031] Post Processed—The ‘Intersection’ frame or FIG. 12 shows light areas only when both delta frames are lit. Notice that the bright light to the sides of the model are not visible in this view. To see why both edges of the model are lit, look at the North Delta frame. The right edge of the model gets lit by the North projector, and the left edge of the model gets lit by the South projector—on the far side of the ring. The same thing happens for the East Delta frame and the intersection of these two allows the extents of the model to be extracted.

[0032] Lens calibration preferably occurs once—during manufacturing—and it stores the characteristics of the individual lens into the EEPROM memory onboard the camera. A combination of the lens and field calibration allows the program to pair a camera pixel with an accurate three dimensional ray exiting the camera lens. Field calibration may be performed by the user at any time. It corrects for any shift or rotation of the camera card, or a slight deformation of the ring. This calibration uses a picture of the blue lights on the far side of the ring, as they are at a known physical position in space. A combination of the lens and field calibration allows the program to pair a camera pixel with an accurate three dimensional ray exiting the camera lens. Projector calibration may be performed by the user at any time. It determines the location of the planes of light being projected into the ring. This calibration correlates two delta frames from different cameras to determine the position of a calibration target held in front of the projector, normalizes the planes, and stores the result onboard the projector card. The projector calibration allows the program to know the precise mathematical formula of each three dimensional plane of light projected onto a model.

[0033] From a software perspective, capturing a 3D shape using the present invention preferably begins by connecting to the structure, loading the lens, field, and projector calibrations, setting up the cameras, and taking a triple capture on all four cameras simultaneously. At that point, having 12 pictures of the model (ambient/north-lit/east-lit from each of four cameras), and many other possibilities for post-processed images.

[0034] Once the captures are taken, there are six steps to create a model in the preferred embodiment:

• • [0035] 1. Extract Edges from 2D Images. There are eight ‘Delta frames’ (Northeast camera—North-Lit, Northeast camera—East-Lit, Southeast camera—South-lit, etc). For each of these delta frames, identify the projected stripes, and extract the left and right edges of the each stripe. Because this data may be too close together (circumferentially) to be useful, average the right and left edges to create a single ‘edge’ for each stripe. In order not to extract stripes that may be in the background, this routine limits its ‘filed of view’ to the 2D extents of the model, found by examining the Intersection frame.
  • [0036] 2. Correlate 2D Edges with 3D Planes. At this point, there are several edges extracted for each of the eight camera/projector pairs. This routine figures out which 2D edge corresponds to which 3D projected stripe. For instance, if there are three edges extracted for ‘northeast-north-lit’, the edges should correspond to projected ‘north stripes’ number 1/2/3, 2/3/4, . . . , or 18/19/20.
  • [0037] 3. Convert Edges to 3D. Once knowing how each 2D (camera) edge corresponds to a 3D (projector) stripe, it is mathematically possible to convert to 3D edges. Find the intersection of a camera ray with a projector plane.
  • [0038] 4. Merge Overlapping 3D Edges. If the same (projector) stripe is seen by two separate cameras, there will be presented two 3D edges that are identical (except that they may be slightly longer, shorter, etc). This section averages them together and feathers the edges.
  • [0039] 5. Assemble the 3D Model. Using the post position and the array of 3D edges ‘hanging in space’, this routine builds a full Tracer model, smoothes, and completes the distal end and continues.
  • [0040] 6. Extract and Add Landmarks to Model. Once the full model is built, this routine extracts dark dots from the ambient frames for each camera and projects any landmarks that are found onto the model.

[0041] The description herein of the preferred embodiment of the invention is for exemplary purposes and is not intended in any way to limit the scope of the allowed claims. The allowed claims are to be given their broadest interpretation under the law.