Real Time Photogrammetry: Methods and Systems for

Deformation Analysis

(January, 2018)

Shahab Ahmed,

Engineering Photogrammetry, MSc Geomatics, Hochschule Karlsruhe, Technik und

Wirtschaft

Abstract— Today’s technological advancements, speed

of image processing tools as well as the availability of robust techniques,

methods and systems for extracting geometric and basic thematic information

from image streams makes real-time photogrammetry possible. The paper discusses

the touchless measurement methods and systems in real time photogrammetry for

deformation analysis and dynamic process using point cloud and other

acquisition systems. It also discusses processing methods to reconstruct

surfaces for motion tracking and deformation analysis. The example mthods and

systems demonstrate the current research and today’s potential for future

applications.

xxxx-xxxx/0x/$xx.00 © 200x IEEE Published by the IEEE Computer

Society

—————————— u

——————————

1 Introduction

Real-time photogrammetry has been a dream

since its beginning. The technological advancements on the last few years make

the realization of this dream possible. This paper gives an overview about

automated and real time photogrammetry, its methods and systems and its

application in deformation analysis.

Real-Time Photogrammetry, in the sense of Close-Range

Photogrammetry, has reached a high-level of performance. It has evolved from a

research topic to a viable technology for a large number of applications. Significant

advances in a number of areas have led to a vast increase in the performance of

Real-Time Photogrammetric Systems. The resolution, i.e. number of sensor

elements, of CCTV-type (Closed Circuit Television) solid-state cameras has been

improved, pixelsynchronous frame grabbers are widely available, and the storage

and processing capabilities of computers have made dramatic progress. Advances

in the radiometric and geometric characterization and calibration of CCD

cameras made it possible to attain very high accuracies 1.

Photogrammetry as an image correlation

technique determines geometric properties, such as the displacement and strain

history of a deformation event. Real time photogrammetry makes possible to

track the minute changes in the speckle pattern on the area of interest. These

minute changes are then translated into three-dimensional displacement vectors

as a function of time. Important mechanical behavior, such as strain or shear

angle, can be calculated from the displacement vectors. A clear advantage of

real photogrammetric measurement over other, more conventional techniques is

the fast sample rate of the data acquisition. CCD cameras and video systems can

be used very effectively to analyse dynamic objects or cases of rapid

deformation.

Various techniques and systems have been

developed over time to perform photogrammetry in real time according to the

needs of the projects.

2 More

about Real TIme Photogrammetry

2.1 Real Time

Data Capture

The ability to

capture digital data in real time is the essential advantage of CCD cameras

over conventional film based camera systems. Furthermore, the data can be captured

at a relatively fast sampling rate of 25 or 30 frames per second, thereby allowing

the measurement and analysis of changing processes or quick deformations. The

real time capture and fast sample rate has been exploited to great effect by

the machine vision and, more recently, the close range photogrammetry communities.

Both two and three dimensional measurement tasks are carried out routinely over

a wide range of applications. However, unless the measurement is a relatively

straightforward task under controlled conditions, real time processing and

analysis requires more computer processing power than can be made conveniently

available. Although advances in technology are broadening the range of image processing

computations which can be carried out at frame rate, very complicated metrology

applications are beyond the current capabilities of machine vision systems 2.

Figure 1.

Direct CCD image acquisition for real time or near real time processing.

2.2

Camera Calibrations

Photogrammetry

deals with precise measurements from imagesThus; accurate calibration of the

imaging sensors is one of the major goals. The geometrical model of the image

acquisition process in a real time photogrammetric system is a perspective

projection. The bundle method associates the adjustment of the image

measurements and the estimation of the camera parameters. The bundle method is considered

the most flexible, general, and accurate sensor model, widely used in

close-range applications 3.

3 Deformation Analysis

Different photogrammetric processes e.g. Terrestrial laser

scanning (TLS) are being extensively used in the earth sciences to understand

and monitor earth surface properties and processes by making touchless measurements.

It is commonly used to create dense three-dimensional (3-D) point clouds or

digital elevation models to map and characterize the earth surface, and to better

understand surface processes by comparing multiple acquisitions over time.

Dense 3-D data are also used to quantify and characterize natural hazards and

to monitor hazard processes. The use of TLS and other remote sensing

technologies now forms an important part of natural hazard risk management

approaches also 4.

The current state-of-the-art of terrestrial

laser scanning (TLS) provides several solutions for the automated surface

reconstruction of a wide variety of objects. Comparison of two or more point

clouds gathered at different epochs potentially enables the detection of

variations on the surface geometry and position. These variations can be

grouped into three main categories: (a) rigid-body transformations, (b) shape

changes, i.e., modifi cations of the surface without loss of material that

result in deformations; i.e., variations of the ratio between displacement and

relative distance between two points, and (c) changes related to loss or

deposition of material. Traditional surveying techniques adopted for the

evaluation of such phenomena are based on precise measurement of a few control

points. On the other hand, the comparison of laser scanning point clouds

introduces the opportunity to operate a deformation analysis on a wider area,

although with a lower resolution of detectable variations than with

single-point observations. This kind of analysis also allows for change

detection, i.e., the investigation of variations of type (c) 5.

The key challenges in using TLS to study

earth processes at the super-temporal level is the high cost of frequent data

acquisitions and challenges in processing and managing large numbers of data.

The advent of automated terrestrial laser scanners (ATLSs) has made

high-temporal-resolution terrestrial acquisitions easier; however, automatic

processing of the data is still required to relieve the post-processing burden.

This is especially important for monitoring systems, where processed results

are needed as soon as possible for decision makers.But automation of all this data

acquisition, surface reconstruction, detecting displacements and deformation in

real time or near real time is a big challenge and a few studies have been

carried out in this regard to develop methods and systems.

4 Methods And Systems

4.1 Real-time

change detection using automated terrestrial laser scanning

R. A. Kromer et al 5 showed in

their study about building a real time system for automated terrestrial laser

scanning with near-real-time change detection.

Figure 4.

Near-real-time data processing workflow consisting of a data automated

acquisition module, a pre-treatment and point cloud stage, a rejection pipeline

consisting of an initial alignment and an iterative fine alignment stage, a 4-D

filteing and distance calculation algorithm (Kromer et al., 2015b) and a

visualization module. This workflow is repeated for each point cloud

acquisition.

Optech ILRIS long-range (LR) laser scanner

was used for data acquisition and then two filters were applied to the data, a statistical

outlier removal and a pass through filter, available in the PCL filter. The

statistical outlier removal was used to remove areas with low point densities

and sparse outliers, such as artefacts from multipath or edge effects. By

removing these points, errors in calculating surface normals, in registering

the point cloud and in change detection are reduced. The outlier removal

algorithm calculates for each point the distance to all its neighbours and

removes points whose distances are outside of the point cloud’s global mean and

standard deviation. The pass through filter is used to remove points outside of

a specified target area. For example, these may include points in the foreground

or background or densely vegetated areas. This is done by defining limits in

each dimension where points falling outside are to be removed. The next

pre-treatment step is querying the total number of points acquired in the point

cloud. If the number of points does not meet a pre-defined threshold, the

entire point cloud is rejected, no output is generated and the processing is

queued until the next point cloud is intercepted.

Registration pipeline to consist of two

main steps, an initial alignment stage and a fine alignment stage using the PCL

registration application programming interface. The purpose of this registration

pipeline and its design was to improve overall convergence time of the registration

and to align clouds that are far apart, in cases where the scanner was moved.

Using this above defined near real time

photogrammetric system the a change detection method called a four-dimensional

(4-D; space and time) algorithm to detect change between successive point

clouds and filter random noise due to surface roughness and instrumental error

using neighbourhood distance values in both space and time. We apply an

empirical calibration step to subtract systematic errors that are a result of

using the same reference scan for all distance calculations from the reference

scan. Point cloud to point cloud distances are averaged using neighbourhood

distance values in space and through time. A balance between spatial and

temporal averaging should be optimized for the signal being studied, to avoid

spatial or temporal smoothing of the distance values. The combined total of

spatial and temporal neighbours for averaging also determines the reduction in

uncertainty of the calculated mean distance values.

Finally the data visualization was done

using basic point cloud visualizer using the PCL’s visualization class 5.

This study reveals the method and related

system for touchless deformation analysis using a real time photogrammetric

system.

4.2 A robust real-time surface

reconstruction method on point clouds captured from a 3D surface photogrammetry

system

The

real time photogrammetric systems are further based on methods and algrorithms

for processing data in such a way so that they are adapted in real time or near

real time systems and are also capable of detecting dynamic processes such as

deformations and animations etc. Wenyang Liu et al 6 proposed a robust real-time

surface reconstruction method on point clouds captured from a 3D surface

photogrammetry system. In this research a robust

and fast surface reconstruction method was developed on point clouds acquired

by the photogrammetry system in medical physics, without explicitly solving the

partial differential equation (PDE) required by a typical variational approach.

Instead the reasearchers took advantage of the overcomplete nature of the

acquired point clouds, and they proposed a method which solves and propagates a

sparse linear relationship from the point cloud manifold to the surface

manifold, assuming both manifolds share similar local geometry, so that the

deformations can be detected. By relatively continuous and consistent point

cloud acquisitions, the researchers propose a sparse regression (SR) model to

directly approximate the target point cloud as a sparse linear combination from

the training set, assuming that the point correspondences built by the

iterative closest point (ICP) is reasonably accurate and have residual errors

following a Gaussian distribution. The authors quantitatively evaluated the

reconstruction performance with respect to root-mean-squared-error, by

comparing its reconstruction results against that from the variational method 6.

2.?METHOD

The reconstructed surface

should have a representation and data structure that is not only good for

static rendering but should also be good for deformation and dynamic operations

on surfaces. None of the present approaches possess all of these properties. In

general there are two kinds of surface representations, explicit or implicit.

Explicit surfaces prescribe the precise location of a surface while implicit surfaces

represent a surface as a particular isocontour of a scalar function. Popular

explicit representations include parametric surfaces and triangulated surfaces.

Following methods describe the reconstruction and deformation detection of the

surfaces 7.

2a. Variational surface

reconstruction method

Here the level set

method as a powerful numerical technique for the deformation of implicit

surfaces was used. Although implicit surfaces have been used in computer

graphics for quite a while, they were mostly used for static modeling and were based

on discrete formulations 7. The level set method is based on a continuous

formulation using PDEs and allows one to detect deformation of an implicit

surface. The level-set method is especially suitable for point cloud

representation, as it provides a continuous representation of underlying surfaces

and avoids constructing explicit point correspondences for capturing and

tracking deformations. Specifically, given a surface S, it is implicitly

represented as the zero level-set of its corresponding level-set function in a

space that is one dimension higher. Provided a point cloud p, by optimizing the

following regularized functional the level-set surface is constructed:

E(?(x)) = ??d(x, p)?(?(x))|??(x)|dx, (1)

where

d(x, p) defines a distance function to the point cloud p, ?(x) represents the

level-set function, and ? is the Dirac delta function. Equation (1) is usually

solved with gradient descent by evolving the following PDE:

???t=????????(?d???|??|+d????|??|) (2)

This PDE is takes time

to solve and hence it is slow, so an alternative fast and robust surface

reconstruction method is required to meet the real-time requirement, where a

subsecond reconstruction time is desirable for point clouds to construct them

real time and detect deformations 6.

2b. Proposed fast surface

reconstruction method

In their study Wenyang

Liu et al 6 proposed their own robust but fast enough surface reconstruction

method, taking advantage of the repetitive and overcomplete nature of the

acquired point cloud set and using the existing variational method. Given

sufficient number of training point clouds and their corresponding level-set

surfaces, the target point cloud as a sparse linear combinations from the

training set were approximated.

First the training

level-set surfaces from the training point clouds were reconstructed, following

steps from previously proposed variational reconstruction method e.g. given k

training point clouds {p1, p2, …, pk}, their corresponding surfaces as {?1, ?2, …, ?k}, with ?i ? ?N being the level-set

function representing each reconstructed surface were reconstructed. This step

requires the most computational efforts and is performed offline.

The iterative closest

point (ICP) algorithm is used to align the training point clouds. Specifically,

a point cloud p1 ? ?n in the training set was arbitrarily chosen as the

reference and the rest of the point clouds into the same coordinate system with

correspondences using ICP were transformed. The target point cloud were then

represented through sparse linear regression. Supposing that training point

clouds and their level-set surfaces share the same topological geometry the

target level-set surface was reconstructed. Finally, the target surface is transformed

back to its original coordinates depicting deformations detected by level set

method.

This study focused on

the robust surface reconstruction

method on point clouds captured from a 3D surface photogrammetry system and detecting deformations in real time.

4.3 High-speed photogrammetry Method

There are also other methods in the

market other than point clouds which use real time photogrammetric system to

investigate dynamic deformation. In the study conducted by by Jian H. Yu et al

8, a unique high-speed imaging technique was used which involves the use of

two high-speed cameras to record stereo images of a speckle patterned impact

area and subsequent photogrammetric analysis. Using an image correlation technique

in photogrammetry that determines geometric properties, the displacement and

strain history of a deformation event can be tracked, by tracking the small

changes in the speckle pattern on the area of interest. These small changes are

then translated into three-dimensional displacement vectors as a function of

time to keep track of the dynamic deformations. Influential mechanical

behavior, such as strain or shear angle, can be calculated from these three

dimensional displacement vectors. By combining high-speed photography with

photogrammetry, a full-field view on the strain as a function of time is made

possible, and it can be detected as deformation in real time.

5 Conclusion

In this study, a glimpse of real time photogrammetry was given and

the technological advancements in this field were also discussed which made the

realization of such real time systems possible. Some methods and systems have

been reaserched and studied e.g. automated and real time photogrammetry, its

touchless measurement methods using TLS for point cloud acquisition and

de-formation analysis. The reasearch also discusses another robust yet fast

surface reconstruction method in real time in the field of medical science. These

studies focus on measurements of point clouds and resulting surface

reconstructions from these point clouds to detect dynamic processes e.g.

deformations in real time. Techniques other than those based on point clouds also

exist e.g. high speed photogrammetry. These proposed systems and methods of

real time photogrammetry for deformation analysis and dynamic processes bring

this field to next level i.e. evolve it from a research topic to a viable

technology for a large number of applications.

References

Gustafson, P.C., 1991. An Accuracy! Repeatability

Test for a Video Photogrammetric Measurement. SPIE Vol. 1526, Industrial Vision

Metrology, pp. 36 – 41.Mark R. Shortis, “The influence of storage media on the accuracy

and repeatability of photogrammetric measurements using CCD cameras”.

Remondino F., 2004, “3-D Reconstruction

of Static Human Body Shape from Image Sequence, Computer Vision and Image Understanding”.Ryan A. Kromer, “Automated terrestrial

laser scanning with near-real-time change detection – monitoring of the

Séchilienne landslide,” Published: 24 May

2017Marco Scaioni, Riccardo Roncella, and

Mario Ivan Alba, “Change Detection and Deformation Analysis in Point Clouds:

Application to Rock Face Monitoring,” Wenyang Liu et al, “A robust real-time

surface reconstruction method on point clouds captured from a 3D surface photogrammetry

system”, Published online 2016 Apr 13Zhao H.-K.,

Osher S., and Fedkiw R., “Fast surface reconstruction using the level set

method,” in Proceedings of IEEE Workshop on Variational and Level Set Methods

in Computer Vision (IEEE, Vancouver, BC, 2001), pp. 194–201.Jian H. Yu, Alex J. Hsieh, Peter G. Dehmer, and James M. Sands,

“Real-Time Full-field Deformation Analysis on the Ballistic Impact of Polymeric

Materials Using High-speed Photogrammetry”