Real advancements, speed of image processing tools as well

Real Time Photogrammetry: Methods and Systems for
Deformation Analysis

(January, 2018)

We Will Write a Custom Essay Specifically
For You For Only $13.90/page!

order now

Shahab Ahmed,
Engineering Photogrammetry, MSc Geomatics, Hochschule Karlsruhe, Technik und


Abstract— Today’s technological advancements, speed
of image processing tools as well as the availability of robust techniques,
methods and systems for extracting geometric and basic thematic information
from image streams makes real-time photogrammetry possible. The paper discusses
the touchless measurement methods and systems in real time photogrammetry for
deformation analysis and dynamic process using point cloud and other
acquisition systems. It also discusses processing methods to reconstruct
surfaces for motion tracking and deformation analysis. The example mthods and
systems demonstrate the current research and today’s potential for future

xxxx-xxxx/0x/$xx.00 © 200x IEEE        Published by the IEEE Computer

——————————   u  

1   Introduction

Real-time photogrammetry has been a dream
since its beginning. The technological advancements on the last few years make
the realization of this dream possible. This paper gives an overview about
automated and real time photogrammetry, its methods and systems and its
application in deformation analysis.

Real-Time Photogrammetry, in the sense of Close-Range
Photogrammetry, has reached a high-level of performance. It has evolved from a
research topic to a viable technology for a large number of applications. Significant
advances in a number of areas have led to a vast increase in the performance of
Real-Time Photogrammetric Systems. The resolution, i.e. number of sensor
elements, of CCTV-type (Closed Circuit Television) solid-state cameras has been
improved, pixelsynchronous frame grabbers are widely available, and the storage
and processing capabilities of computers have made dramatic progress. Advances
in the radiometric and geometric characterization and calibration of CCD
cameras made it possible to attain very high accuracies 1.

Photogrammetry as an image correlation
technique determines geometric properties, such as the displacement and strain
history of a deformation event. Real time photogrammetry makes possible to
track the minute changes in the speckle pattern on the area of interest. These
minute changes are then translated into three-dimensional displacement vectors
as a function of time. Important mechanical behavior, such as strain or shear
angle, can be calculated from the displacement vectors. A clear advantage of
real photogrammetric measurement over other, more conventional techniques is
the fast sample rate of the data acquisition. CCD cameras and video systems can
be used very effectively to analyse dynamic objects or cases of rapid

Various techniques and systems have been
developed over time to perform photogrammetry in real time according to the
needs of the projects.


2   More
about Real TIme Photogrammetry

2.1 Real Time
Data Capture


The ability to
capture digital data in real time is the essential advantage of CCD cameras
over conventional film based camera systems. Furthermore, the data can be captured
at a relatively fast sampling rate of 25 or 30 frames per second, thereby allowing
the measurement and analysis of changing processes or quick deformations. The
real time capture and fast sample rate has been exploited to great effect by
the machine vision and, more recently, the close range photogrammetry communities.
Both two and three dimensional measurement tasks are carried out routinely over
a wide range of applications. However, unless the measurement is a relatively
straightforward task under controlled conditions, real time processing and
analysis requires more computer processing power than can be made conveniently
available. Although advances in technology are broadening the range of image processing
computations which can be carried out at frame rate, very complicated metrology
applications are beyond the current capabilities of machine vision systems 2.







Figure 1.
Direct CCD image acquisition for real time or near real time processing.

Camera Calibrations

deals with precise measurements from imagesThus; accurate calibration of the
imaging sensors is one of the major goals. The geometrical model of the image
acquisition process in a real time photogrammetric system is a perspective
projection. The bundle method associates the adjustment of the image
measurements and the estimation of the camera parameters. The bundle method is considered
the most flexible, general, and accurate sensor model, widely used in
close-range applications 3.

3   Deformation Analysis

Different photogrammetric processes e.g. Terrestrial laser
scanning (TLS) are being extensively used in the earth sciences to understand
and monitor earth surface properties and processes by making touchless measurements.
It is commonly used to create dense three-dimensional (3-D) point clouds or
digital elevation models to map and characterize the earth surface, and to better
understand surface processes by comparing multiple acquisitions over time.
Dense 3-D data are also used to quantify and characterize natural hazards and
to monitor hazard processes. The use of TLS and other remote sensing
technologies now forms an important part of natural hazard risk management
approaches also 4.

The current state-of-the-art of terrestrial
laser scanning (TLS) provides several solutions for the automated surface
reconstruction of a wide variety of objects. Comparison of two or more point
clouds gathered at different epochs potentially enables the detection of
variations on the surface geometry and position. These variations can be
grouped into three main categories: (a) rigid-body transformations, (b) shape
changes, i.e., modifi cations of the surface without loss of material that
result in deformations; i.e., variations of the ratio between displacement and
relative distance between two points, and (c) changes related to loss or
deposition of material. Traditional surveying techniques adopted for the
evaluation of such phenomena are based on precise measurement of a few control
points. On the other hand, the comparison of laser scanning point clouds
introduces the opportunity to operate a deformation analysis on a wider area,
although with a lower resolution of detectable variations than with
single-point observations. This kind of analysis also allows for change
detection, i.e., the investigation of variations of type (c) 5.

The key challenges in using TLS to study
earth processes at the super-temporal level is the high cost of frequent data
acquisitions and challenges in processing and managing large numbers of data.
The advent of automated terrestrial laser scanners (ATLSs) has made
high-temporal-resolution terrestrial acquisitions easier; however, automatic
processing of the data is still required to relieve the post-processing burden.
This is especially important for monitoring systems, where processed results
are needed as soon as possible for decision makers.But automation of all this data
acquisition, surface reconstruction, detecting displacements and deformation in
real time or near real time is a big challenge and a few studies have been
carried out in this regard to develop methods and systems.

4    Methods And Systems

4.1  Real-time
change detection using automated terrestrial laser scanning


R. A. Kromer et al 5 showed in
their study about building a real time system for automated terrestrial laser
scanning with near-real-time change detection.
























Figure 4.
Near-real-time data processing workflow consisting of a data automated
acquisition module, a pre-treatment and point cloud stage, a rejection pipeline
consisting of an initial alignment and an iterative fine alignment stage, a 4-D
filteing and distance calculation algorithm (Kromer et al., 2015b) and a
visualization module. This workflow is repeated for each point cloud


Optech ILRIS long-range (LR) laser scanner
was used for data acquisition and then two filters were applied to the data, a statistical
outlier removal and a pass through filter, available in the PCL filter. The
statistical outlier removal was used to remove areas with low point densities
and sparse outliers, such as artefacts from multipath or edge effects. By
removing these points, errors in calculating surface normals, in registering
the point cloud and in change detection are reduced. The outlier removal
algorithm calculates for each point the distance to all its neighbours and
removes points whose distances are outside of the point cloud’s global mean and
standard deviation. The pass through filter is used to remove points outside of
a specified target area. For example, these may include points in the foreground
or background or densely vegetated areas. This is done by defining limits in
each dimension where points falling outside are to be removed. The next
pre-treatment step is querying the total number of points acquired in the point
cloud. If the number of points does not meet a pre-defined threshold, the
entire point cloud is rejected, no output is generated and the processing is
queued until the next point cloud is intercepted.

Registration pipeline to consist of two
main steps, an initial alignment stage and a fine alignment stage using the PCL
registration application programming interface. The purpose of this registration
pipeline and its design was to improve overall convergence time of the registration
and to align clouds that are far apart, in cases where the scanner was moved.

Using this above defined near real time
photogrammetric system the a change detection method called a four-dimensional
(4-D; space and time) algorithm to detect change between successive point
clouds and filter random noise due to surface roughness and instrumental error
using neighbourhood distance values in both space and time. We apply an
empirical calibration step to subtract systematic errors that are a result of
using the same reference scan for all distance calculations from the reference
scan. Point cloud to point cloud distances are averaged using neighbourhood
distance values in space and through time. A balance between spatial and
temporal averaging should be optimized for the signal being studied, to avoid
spatial or temporal smoothing of the distance values. The combined total of
spatial and temporal neighbours for averaging also determines the reduction in
uncertainty of the calculated mean distance values.

Finally the data visualization was done
using basic point cloud visualizer using the PCL’s visualization class 5.

This study reveals the method and related
system for touchless deformation analysis using a real time photogrammetric


4.2 A robust real-time surface
reconstruction method on point clouds captured from a 3D surface photogrammetry


real time photogrammetric systems are further based on methods and algrorithms
for processing data in such a way so that they are adapted in real time or near
real time systems and are also capable of detecting dynamic processes such as
deformations and animations etc. Wenyang Liu et al 6 proposed a robust real-time
surface reconstruction method on point clouds captured from a 3D surface
photogrammetry system. In this research a robust
and fast surface reconstruction method was developed on point clouds acquired
by the photogrammetry system in medical physics, without explicitly solving the
partial differential equation (PDE) required by a typical variational approach.
Instead the reasearchers took advantage of the overcomplete nature of the
acquired point clouds, and they proposed a method which solves and propagates a
sparse linear relationship from the point cloud manifold to the surface
manifold, assuming both manifolds share similar local geometry, so that the
deformations can be detected. By relatively continuous and consistent point
cloud acquisitions, the researchers propose a sparse regression (SR) model to
directly approximate the target point cloud as a sparse linear combination from
the training set, assuming that the point correspondences built by the
iterative closest point (ICP) is reasonably accurate and have residual errors
following a Gaussian distribution. The authors quantitatively evaluated the
reconstruction performance with respect to root-mean-squared-error, by
comparing its reconstruction results against that from the variational method 6.


The reconstructed surface
should have a representation and data structure that is not only good for
static rendering but should also be good for deformation and dynamic operations
on surfaces. None of the present approaches possess all of these properties. In
general there are two kinds of surface representations, explicit or implicit.
Explicit surfaces prescribe the precise location of a surface while implicit surfaces
represent a surface as a particular isocontour of a scalar function. Popular
explicit representations include parametric surfaces and triangulated surfaces.
Following methods describe the reconstruction and deformation detection of the
surfaces 7.


2a. Variational surface
reconstruction method

Here the level set
method as a powerful numerical technique for the deformation of implicit
surfaces was used. Although implicit surfaces have been used in computer
graphics for quite a while, they were mostly used for static modeling and were based
on discrete formulations 7. The level set method is based on a continuous
formulation using PDEs and allows one to detect deformation of an implicit
surface. The level-set method is especially suitable for point cloud
representation, as it provides a continuous representation of underlying surfaces
and avoids constructing explicit point correspondences for capturing and
tracking deformations. Specifically, given a surface S, it is implicitly
represented as the zero level-set of its corresponding level-set function in a
space that is one dimension higher. Provided a point cloud p, by optimizing the
following regularized functional the level-set surface is constructed:


E(?(x)) = ??d(x, p)?(?(x))|??(x)|dx,                   (1)


d(x, p) defines a distance function to the point cloud p, ?(x) represents the
level-set function, and ? is the Dirac delta function. Equation (1) is usually
solved with gradient descent by evolving the following PDE:


???t=????????(?d???|??|+d????|??|)                            (2)


This PDE is takes time
to solve and hence it is slow, so an alternative fast and robust surface
reconstruction method is required to meet the real-time requirement, where a
subsecond reconstruction time is desirable for point clouds to construct them
real time and detect deformations 6.


2b. Proposed fast surface
reconstruction method

In their study Wenyang
Liu et al 6 proposed their own robust but fast enough surface reconstruction
method, taking advantage of the repetitive and overcomplete nature of the
acquired point cloud set and using the existing variational method. Given
sufficient number of training point clouds and their corresponding level-set
surfaces, the target point cloud as a sparse linear combinations from the
training set were approximated.

First the training
level-set surfaces from the training point clouds were reconstructed, following
steps from previously proposed variational reconstruction method e.g. given k
training point clouds {p1, p2, …, pk}, their corresponding surfaces as {?1, ?2, …, ?k}, with ?i ? ?N being the level-set
function representing each reconstructed surface were reconstructed. This step
requires the most computational efforts and is performed offline.

The iterative closest
point (ICP) algorithm is used to align the training point clouds. Specifically,
a point cloud p1 ? ?n in the training set was arbitrarily chosen as the
reference and the rest of the point clouds into the same coordinate system with
correspondences using ICP were transformed. The target point cloud were then
represented through sparse linear regression. Supposing that training point
clouds and their level-set surfaces share the same topological geometry the
target level-set surface was reconstructed. Finally, the target surface is transformed
back to its original coordinates depicting deformations detected by level set

This study focused on
the robust surface reconstruction
method on point clouds captured from a 3D surface photogrammetry system and detecting deformations in real time.


4.3  High-speed photogrammetry Method

There are also other methods in the
market other than point clouds which use real time photogrammetric system to
investigate dynamic deformation. In the study conducted by by Jian H. Yu et al
8, a unique high-speed imaging technique was used which involves the use of
two high-speed cameras to record stereo images of a speckle patterned impact
area and subsequent photogrammetric analysis. Using an image correlation technique
in photogrammetry that determines geometric properties, the displacement and
strain history of a deformation event can be tracked, by tracking the small
changes in the speckle pattern on the area of interest. These small changes are
then translated into three-dimensional displacement vectors as a function of
time to keep track of the dynamic deformations. Influential mechanical
behavior, such as strain or shear angle, can be calculated from these three
dimensional displacement vectors. By combining high-speed photography with
photogrammetry, a full-field view on the strain as a function of time is made
possible, and it can be detected as deformation in real time.


5    Conclusion

In this study, a glimpse of real time photogrammetry was given and
the technological advancements in this field were also discussed which made the
realization of such real time systems possible. Some methods and systems have
been reaserched and studied e.g. automated and real time photogrammetry, its
touchless measurement methods using TLS for point cloud acquisition and
de-formation analysis. The reasearch also discusses another robust yet fast
surface reconstruction method in real time in the field of medical science. These
studies focus on measurements of point clouds and resulting surface
reconstructions from these point clouds to detect dynamic processes e.g.
deformations in real time. Techniques other than those based on point clouds also
exist e.g. high speed photogrammetry. These proposed systems and methods of
real time photogrammetry for deformation analysis and dynamic processes bring
this field to next level i.e. evolve it from a research topic to a viable
technology for a large number of applications.


Gustafson, P.C., 1991. An Accuracy! Repeatability
Test for a Video Photogrammetric Measurement. SPIE Vol. 1526, Industrial Vision
Metrology, pp. 36 – 41.Mark R. Shortis, “The influence of storage media on the accuracy
and repeatability of photogrammetric measurements using CCD cameras”.
Remondino F., 2004, “3-D Reconstruction
of Static Human Body Shape from Image Sequence, Computer Vision and Image Understanding”.Ryan A. Kromer, “Automated terrestrial
laser scanning with near-real-time change detection – monitoring of the
Séchilienne landslide,” Published: 24 May
2017Marco Scaioni, Riccardo Roncella, and
Mario Ivan Alba, “Change Detection and Deformation Analysis in Point Clouds:
Application to Rock Face Monitoring,” Wenyang Liu et al, “A robust real-time
surface reconstruction method on point clouds captured from a 3D surface photogrammetry
system”, Published online 2016 Apr 13Zhao H.-K.,
Osher S., and Fedkiw R., “Fast surface reconstruction using the level set
method,” in Proceedings of IEEE Workshop on Variational and Level Set Methods
in Computer Vision (IEEE, Vancouver, BC, 2001), pp. 194–201.Jian H. Yu, Alex J. Hsieh, Peter G. Dehmer, and James M. Sands,
“Real-Time Full-field Deformation Analysis on the Ballistic Impact of Polymeric
Materials Using High-speed Photogrammetry”