Post-processing panoramic imagery geo-rectification system

[0001] This application claims priority to U.S. Provisional Patent Application entitled “POST-PROCESSING PANORAMIC IMAGERY GEO-RECTIFICATION SYSTEM,” Ser. No. 63/399,973, filed Aug. 22, 2022, the disclosures of which are hereby incorporated entirely herein by reference.

[0002] This invention relates generally to an image processing system and more particularly to a post-processing panoramic imagery geo-rectification system for aerial imagery.

[0003] Aerial photography is useful for many different purposes. For example, aerial photography is used for cartography, land-use planning, archaeology, movie production, environmental studies, surveillance, commercial advertising, conveyancing, artistic projects, and many other purposes. Various businesses and entities are utilizing aerial imagery in order to market and provide better service to customers and the like by using GIS maps and the like. However, the metadata associated with the imagery captured by the cameras typically have errors that require post-processing systems to solve or correct. Correction of the camera position for the imagery, along with the addition of warp-defining metadata, often involves manual or human interaction. The imagery that is captured is often compared to a geographic information system (“GIS”) to provide additional information. Existing systems lack automation in the post-processing of the imagery.

[0004] Accordingly, there is a need for a post-processing panoramic imagery geo-rectification system for use with aerial spherical imagery.

[0005] The present invention relates to a post-processing panoramic imagery geo-rectification system for aerial spherical imagery. The system processes panoramic images using at least two subprocesses, wherein one subprocess is a multi-step process to determine or solve errors in the metadata associated with captured images, such as, but not limited to, solving six degrees of freedom for camera position, without human intervention, wherein the six degrees of freedom are latitude, longitude, altitude, pan, tilt, and roll. The system then may implement a second subprocess that is a two-step process to perform image registration in order to eliminate other distortions, such as elevation offsets and stitching errors.

[0006] An embodiment of a post-processing panoramic imagery geo-rectification system for use with aerial imagery includes a server having a processor and a memory and a software application providing instruction to the server to perform the geo-rectification. The system may include a computer having a processor and a memory coupled to the server, wherein a user can access the server through the computer in order to process the imagery.

[0007] The foregoing and other features and advantages of the present invention will be apparent from the following, more detailed description of the particular embodiments of the invention, as illustrated in the accompanying drawings.

[0010] As discussed above, embodiments of the present invention relate to a post-processing panoramic imagery geo-rectification system for aerial imagery. The system processes panoramic imagery and its metadata using at least two subprocesses, as shown in FIG. 1, wherein subprocess 1 is a multi-step process to solve six degrees of freedom for camera position, without human intervention, the six degrees of freedom including latitude, longitude, altitude, pan, tilt, and roll. The system then may implement subprocess 2, which is a two-step process to perform image registration in order to eliminate other distortions, such as elevation offsets and stitching errors.

Subprocess 1: Solving for True Camera Position

Step 1a: Solving for Tilt and Roll Using Horizon Detection

[0011] A machine-learning model is trained to recognize the horizon in a panoramic, equirectangular image. The model is insensitive to pan offsets. The model is used to predict offset values for tilt and roll that visually “level off” the horizon, similar to what a reasonable operator of a 3D viewer would recommend. Generally, the model is accurate within about 0.1 degree more than 90% of the time.

Step 1b: Approximating a Rough Solution for Pan Using the Sun's Azimuth

[0012] A machine-learning model is trained to detect the sun's azimuth in a panoramic, equirectangular image. The model is insensitive to the sun's height in the sky. A software application takes in the time of day and longitude of image capture, predicts the azimuth of where the sun should be in the image, compares that to the output of the trained model, and uses the sun offset to predict a pan offset. The model is accurate within about 2.0 degrees at least 90% of the time.

[0013] Step 1c: Refining Pan Solution Using Fast Fourier Transform (FFT)

• • [0014] 1.c.1) A snapshot image is produced from imagery, pointing directly under the point of image capture (nadir or lowest point). This will be referred to as subprocess 1 snapshot 1.
  • [0015] 1.c.2) Another snapshot of similar size and zoom-level is obtained from a ground-truth map provider. This will be referred to as subprocess 1 snapshot 2.
  • [0016] 1.c.3) Subprocess 1 snapshots 1 and 2 are rotated against each other in increment, 1 degree at a time, not exceeding the 60-degree region of rotation surrounding the pan offset predicted by step 1b. The pixels of each increment are run through a fast Fourier transform (FFT) to determine the similarity of the snapshots. Each 1-degree increment is rated, and the best is chosen.
  • [0017] 1.c.4) Step 1.c.3 is then repeated for a 2-degree region of rotation surrounding the new pan offset estimate, except the rotation is incremented by 0.1 degree, rather than 1 degree. The best similarity between the 2 snapshots, as produced by the same FFT algorithm, becomes the final pan offset prediction.
    Step 1d: Solving for Latitude, Longitude, and Altitude Using Perspective-n-Point (PnP) Pose Computation Algorithm (solvePNP)

[0018] Camera pose offsets are predicted, using an open-source implementation of the Perspective-n-Point (PnP) pose computation algorithm (solvePNP). The tilt, roll, and pan offsets obtained in the previous steps are fed into the solvePNP algorithm, as doing so provides an answer more often, and also provides a better answer on average. The predictions for tilt, roll, and pan produced by solvePNP are discarded and the latitude, longitude, and altitude offset predictions are used.

Subprocess 2: Image Registration, Using Registration Points

[0019] The next 2 steps produce warp points. Warp points, as used herein, define a thin-plate-spline (TPS) server-side, so that when imagery is accessed by a viewer, the viewer provides accurate registration to ground truth.

Step 2a: Digital Elevation Module (DEM)

[0020] A grid of points is defined, approximately 1 mile square, positioned directly under the point of image capture (nadir). For each of those points, elevations are obtained from a ground-truth map provider, producing a grid of elevation points. The elevation points are stored, in memory, to produce parallax offsets for each warp point created by the next step. Once a sphere is fully processed, calculations are used to convert the elevation points into warp points, which are then uploaded to the server.

Step 2b: Using Machine Learning to Place Registration Points

[0021] A machine-learning model is trained, using the RAFT algorithm, to recognize and match common outdoor objects in a pair of standard orthographic-projection map images.

[0022] During inference, a snapshot image is produced from each set of imagery, pointing directly under the point of image capture (nadir). This will be referred to as subprocess 2 snapshot 1. Another snapshot of similar size and zoom-level is obtained from a ground-truth map provider. This will be referred to as subprocess 2 snapshot 2. Publicly sourced latitude/longitude points of street addresses are used to define regions of interest in subprocess 2 snapshot 1. For each region of interest, the model is used to visually seek for some real-world object that is similar between the snapshots. This produces a registration point for that region of interest, which is then uploaded to the server.

[0023] Once the spherical imagery is geo-rectified, it may be utilized by others to access the various geo-references and view particular areas with a greater amount of information.

[0024] It should be appreciated that in addition to geo-rectification of the spherical aerial imagery, the system is also able to determine proper elevation calculations. Thus, if there is a house on a mountain or hill, if the system selects an identifiable portion of the house or near surrounding area as a GCP in both the reference imagery and spherical aerial image, the system not only determines the latitude and longitude in spherical aerial image but determines a proper elevation for that latitude and longitude as well. This happens because of the thin plate spline warp that the system creates for its spherical aerial imagery.

[0025] The embodiments and examples set forth herein were presented in order to best explain the present invention and its practical application and to thereby enable those of ordinary skill in the art to make and use the invention. However, those of ordinary skill in the art will recognize that the foregoing description and examples have been presented for the purposes of illustration and example only. The description as set forth is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the teachings above without departing from the spirit and scope of the forthcoming claims.