JSSSJournal of Sensors and Sensor SystemsJSSSJ. Sens. Sens. Syst.2194-878XCopernicus PublicationsGöttingen, Germany10.5194/jsss-7-13-2018Temperature reconstruction of infrared images with motion deblurringOswald-TrantaBeatebeate.oswald@unileoben.ac.atInstitute for Automation, University of Leoben, Leoben 8700, AustriaBeate Oswald-Tranta (beate.oswald@unileoben.ac.at)11January201871132030August201713November201716November2017This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/This article is available from https://jsss.copernicus.org/articles/7/13/2018/jsss-7-13-2018.htmlThe full text article is available as a PDF file from https://jsss.copernicus.org/articles/7/13/2018/jsss-7-13-2018.pdf
Infrared images of an uncooled microbolometer camera can show significant
blurring effects while recording a moving object. The electrical signal in
the pixel of a microbolometer detector decays exponentially; hence, the
moving object is mapped to more pixels resulting in a blurred image. Not only
the contrast is corrupted by the motion, but also the temperature of the
object seems to be significantly lower. In this paper, it is shown how such
images can be deblurred and the true temperature with a good approximation
restored. Since the detection mechanism of a microbolometer camera is
different from complementary metal–oxide–semiconductor (CMOS) or
charge-coupled device (CCD) cameras, also the point-spread function (PSF)
needed for the deblurring restoration is different. It is shown how the
exponential coefficient of the PSF can be calculated if the motion speed and
the camera resolution are known, or otherwise how it can be estimated from
the image itself. Experimental examples are presented for motion deblurring
used to restore images with linear or rotational motion.
Introduction
The elimination of motion blurring in complementary metal–oxide–semiconductor (CMOS) and charge-coupled device (CCD) camera images is an
extensively investigated topic (Rajagopalan and Chellappa, 2014). Even in the last
years, much research has been inspired by problems, e.g. incidental shake in
hand-held cameras.
Assuming only one motion direction, the problem can be reduced to a simple
image deconvolution. If the blurring kernel (the point-spread function – PSF)
is known, then this is called a non-blind deconvolution. On the other hand,
if the PSF is not known, it has to be estimated first from the image itself,
which is called blind deconvolution. Additional techniques, e.g. Wiener
filter, have been developed for suppressing the noise, which could be
strongly amplified by the deconvolution itself (Rajagopalan and Chellappa, 2014;
Gonzalez et al., 2009).
Motion blurring can be also observed in images of infrared cameras, as it is
described also by Vollmer and Möllmann (Vollmer and Möllmann, 2010). Cooled
cameras with high sensitive photonic detectors have usually a very short
integration time that is typically 1–1.5 ms for room temperature
measurements, and the motion blurring can be mostly neglected. In contrast,
the electrical signal in the pixel of a microbolometer camera decays with a
time constant of 10–15 ms; therefore, a significant blurring effect can be
observed by recording moving objects.
The PSF of motion blurring in a CMOS camera is mainly a linear function with
a necessary length and direction (Rajagopalan and Chellappa, 2014; Gonzalez et al.,
2009). In contrast, the signal of a microbolometer camera decays with an
exponential function, and therefore it requires a PSF with an exponential
decay (Oswald-Tranta et al., 2010a).
Furthermore, by the deblurring of a CMOS camera image, the main goal is to
obtain an image with high contrast, while the recorded value of each pixel
itself has no significance. In contrast, by restoring infrared images, it is
also expected that the recorded infrared (IR) radiation for each pixel and, in a further
step, the correct temperature of the objects should be also reconstructed.
In this paper, it is investigated how in infrared images of a microbolometer
camera the motion distortion can be deblurred. Part of the results have
been published earlier (Oswald-Tranta et al., 2010a; Oswald-Tranta, 2012, 2017).
Here, investigations are shown on how well the temperature
of moving objects can be restored. Different experimental results for the
usage of image deblurring are presented, and it is shown how non-blind and
blind deblurring can be carried out.
Motion blurring in microbolometer images
Experimental setup in the laboratory with a microbolometer and with
a cooled photon detector, recording objects in motion on the conveyor belt.
Blurring of an image is described generally by the equation:
g=h⊗f+n,
where f is the true image recorded under perfect conditions and h is the
so-called PSF representing the distortion of the
acquisition. The convolution of these two (h⊗f) results in the
distorted image. n denotes the additional noise and g the raw image
recorded by the camera. The PSF of a microbolometer camera for a moving
object is an exponential function (Oswald-Tranta et al., 2010a; Oswald-Tranta,
2012):
h=1τime-xτim,for0≤x,otherwise0.
The decay factor τim depends on the relative speed between
camera and the moving object (v), and further on the resolution (r) of
the recorded image given in pixel mm-1. The product vr
represents the speed in pixel s-1. τim depends also on the exponential decay constant
of the electrical signal of the microbolometer detector (τcamera) which has a typical value of about 10 ms:
τim=τcameravr.
In this way, τim represents approximately the number of pixels
that the object moves during the recording time of one image. As cameras with
cooled semiconductor detectors usually work with an integration time of
1–2 ms, τim is about 1 order of
magnitude less than for microbolometer cameras; hence, motion blurring of the
images is in most of the cases negligibly small. However, for microbolometer
cameras, the distortion of the images is often recognisable but can be well
restored by the method described here.
If the temperature difference between object and background is denoted by
ΔTtrue, then the temperature difference in the blurred
image (ΔTblurred) has the following form (Oswald-Tranta et
al., 2010a):
ΔTblurred=0,forx<0ΔTtrue1-exp-xτim,for0≤x≤aΔTtrueexp-x-aτim-exp-xτim,fora<x,
where a denotes the size of the object in pixels. Introducing
m=aτim=dτcamerav,
where d is the real size of the object, the blurred temperature at x=a is
equal to
ΔTblurred=ΔTtrue1-exp-m.
Due to the blurring, small objects with high speed therefore seem to have
much lower temperatures than they really have, because the recorded infrared
radiation is smeared over more pixels during the object motion.
Microbolometer images of a small plate moving from the left to the
right with different speed: 0.3 (a), 0.5 (b), 1 (c) and 1.5 m s-1(d).
Temperature profiles through the blurred and the reconstructed images
of Fig. 2. The corresponding data are summarized in Table 1.
Data of the four measurements in Figs. 2 and 3 are summarized in
the table. For the reconstruction, k=0.005 has been used. The resolution was
r=3 pixel mm-1 and d=10 mm).
In the experiments to test motion blurring, a small conveyor belt is moving
at a specified speed up to 1.5 m s-1 in the field of view of the
camera (see Fig. 1). For comparison, two cameras have been used, a
microbolometer camera (left side) (DIAS, 2011) and a camera with a cooled indium antimonide
(InSb) detector (right side) (FLIR, 2010). For the cooled camera, one may select the
integration time from a wide range; for the experiment, it was set to 1.5 ms,
which allows temperature measurements up to 70 ∘C. Because of the
short integration time, the blurring due to motion is negligible in the images
of this camera.
A small warm plate (65 × 10 × 3 mm3) has been moved
with different speeds in the field of view of both cameras. Table 1
summarizes the data of four measurements, and Fig. 2 shows the corresponding
microbolometer images. It is well visible that the higher the speed, the
more the image is blurred.
Deblurring of the images
It is well known that the Fourier transformation changes the convolution
into a multiplication of the spectra. Therefore, if G, H, F and N
denote the Fourier transformation of the functions g, h, f and n,
then Eq. (1) can be written as
G=H⋅F+N.
In order to obtain an image f̃ close to the true image, the inverse
Fourier transformation of G/H has to be calculated:
f̃=F-1H⋅F+NH=F-1F+NH.
If there were no noise, then f̃=f, which means the image could be
perfectly restored. H, the Fourier transformation of the PSF, is the
so-called optical transfer function (OTF) and it can be calculated from
Eq. (2) (Oswald-Tranta et al., 2010a):
H=11+ipτim,
where p is the frequency. Since H becomes small for high frequencies, the
high-frequency noise gets strongly amplified. Therefore, a kind of low-pass
filter is necessary for the deblurring. A good possibility is to use a
parametrized Wiener filter (Gonzalez et al., 2009; Oswald-Tranta et al.,
2010a):
f̃=F-1H2H2+k⋅F+NH,
where k is a small non-negative number. If k=0, then Eq. (10) is reduced
to Eq. (8).
Amplification factor depending on the frequency due to deconvolution
with the parametrized Wiener filter.
Infrared image of a hot ball, moving with 1 m s-1(a); images restored
with different k values: 0 (b), 0.005 (c) and 0.05 (d).
In Fig. 3, temperature profiles across the four blurred images are shown. Also
the reconstructed temperature profiles are displayed in the diagrams, where
k was set to 0.005. The temperature of the plate was not the same for each
measurement as it was slowly cooling down from one measurement to the next
one. However, the reconstructed temperature values have been compared with the
temperatures measured simultaneously with the cooled IR camera and a very
good correspondence has been found. In Table 1, the observed ΔTblurred/ΔTtrue ratios are compared with
1-exp-m; see also Eq. (6). For the speeds of 1 and
1.5 m s-1, the blurred temperature is significantly less than the true
temperature, but it is very well reconstructed with the proposed Wiener
filter, giving the same value as was measured by the cooled IR camera.
Nevertheless, the higher the speed, the more enhanced the noise after the
reconstruction, which results in an error in the reconstruction.
Temperature profiles through the three restored images compared with
the original blurred one in Fig. 5.
Influence of the Wiener filter parameter
A signal with the frequency p after dividing by the OTF and after the
inverse Fourier transformation is amplified by
f̃f=1H=1+ipτim=1+p2τim2.
Since, according to the Nyquist theorem, the shortest wavelength has to have
at least the length of two pixels, the highest frequency after Fourier
transform of a digital image is p=π. If, e.g. the motion speed v=1 m s-1 and the resolution r=4 pixels mm-1, then
τim=40; thereby, if k=0, the highest frequency noise is
amplified by 1+p2τim2≈πτim≈125.
Using the parametrized Wiener filter, the amplification of a signal with the
frequency p is
A=f̃f=H2H2+k⋅1H=1+p2τim21+k1+p2τim2.
If pτim≫1,
A=f̃f≈pτim1+kp2τim2.
Figure 4 demonstrates this behaviour for τim=40; the larger the
value of k is, the stronger the high-frequency signals are damped. For small
k values, the signal is amplified. If k=1/(πτim)=0.008, then for the highest frequency p=π the amplification A=1.
As it has been shown (Oswald-Tranta et al., 2010a), also the restored
temperature depends on the k value:
ΔTrestoredΔTtrue=1-exp-m21+kk,
where ΔTrestored=Trestored,object-Trestored,background and ΔTtrue=Ttrue,object-Ttrue,background. The larger the k
value is, the lower the reconstructed temperature. With large k values, not
only the high-frequency noise is damped but also the high-frequency
components at the edges of an object; thus, the object becomes smeared and its
reconstructed temperature is lower than the true one.
This is demonstrated in Figs. 5 and 6. A small warm ball with a diameter
of 7 mm was recorded during a motion with 1 m s-1 (see Fig. 5a). The
image is strongly blurred and shows a typical comet-like shape. The
resolution is 4.4 pixel mm-1 and the images are reconstructed with
τim=44. According to the simultaneously recorded images, by the
cooled camera, the temperature of the ball is about 45 ∘C.
Images restored from Fig. 5a using different τim
values: 22 (a) and 66 (b).
If k=0, then the image is very noisy; the high-frequency noise is
amplified by approximately 2 orders of magnitude (Figs. 5b and 6).
If k=0.005, the noise amplification is well suppressed due to the Wiener
filtering, the deblurred image is sharp and the temperature is well restored
(Figs. 5c and 6), which corresponds well with the result of Eq. (14): ΔTrestored/ΔTtrue=0.99.
If k=0.05, then the edges of the restored image are smoothed out due to the
strong damping of high frequencies, causing an elongation of the image
(Figs. 5d and 6). According to Eq. (14), ΔTrestored/ΔTtrue=0.8, which corresponds very well with the measured data.
To our experience, k values around 0.005 are a good compromise. If k is
too low, then the restored image is too noisy. On the other hand, if it is
too high, then due to its low-pass filtering the edges of the object become
smoothed, causing a blurring in the motion direction, which consequently
results also in an apparently lower temperature.
Influence of the deviation in τim
The same image as shown in Fig. 5a has been restored with different τim values (see Figs. 7 and 8). If τim is too low,
then the image is not fully deblurred and a slight distortion remains in the
restored image (see Figs. 7a and 8). On the other hand, if τim is too high, then a kind of negative “shadow” appears behind the object
(see Figs. 7b and 8). Since the deconvolution keeps the total intensity of
the whole image, the restored temperature with a too-low τim value is also too low, and with a too-high one the restored temperature
becomes too high. It is important to note that the used τim values in
Figs. 7 and 8 are 50 and 150 % of the correct value. The ratio of the
restored temperature to the true one is approximately (Oswald-Tranta et al.,
2010a; Oswald-Tranta, 2012)
ΔTrestoredΔTtrue=1-τim-τ2τim⋅exp-m,
where τ2 is the used coefficient, instead of the correct τim. For the example of Figs. 7 and 8, with a=30 pixels,
τim=44, τ2=0.5τim and ΔTtrue=22∘C results in a temperature of about ΔTrestored≈0.75⋅ΔTtrue≈0.75⋅22∘C ≈ 16.5 ∘C. On the other hand, if
τ2=1.5τim, then ΔTrestored≈1.25⋅ΔTtrue≈1.25⋅22∘C ≈ 27.5 ∘C. If the used coefficient is
closer to the correct value, e.g. τ2=0.9τim,
then the deviation of the reconstructed temperature from the true one would
be about 1.1 ∘C. On the other hand, it is also important to note that only
small objects with high speed lose temperature due to incorrect deblurring;
large objects can be well restored even if τim is not exactly
known.
Temperature profiles through the three restored images of Figs. 7 and
5c compared with the original one in Fig. 5a.
Sharpness of restored images
In many cases, e.g. in thermographic non-destructive testing, non-calibrated
cameras are used, because the temperature value itself is not important but only
its distribution and a good contrast in the images (Maldague, 2001;
Oswald-Tranta et al., 2010b; Oswald-Tranta, 2012).
Figure 9a and b demonstrates how well the sharpness of an image can be
restored with the proposed deblurring algorithm. A 10 × 10 cm2
metallic square ruler has been moved with the conveyor belt with
1 m s-1 speed in the field of view of the camera. Due to their
different emissivity, the digits can be well read in a static infrared image,
but after motion distortion they cannot be recognized anymore. However, after
reconstruction, the digits become readable again (see Fig. 9b). It is important to note
that the object in Fig. 9 has a square shape, but the recorded and also the
restored images are squeezed.
A square-shaped metallic ruler during the motion (a), the deblurred
image (b) and the deblurred image applying also the shearing transformation
of Eq. (17) (c).
Distortion due to continuous read-out
As the microbolometer camera is not a snapshot camera and it reads out the
rows of pixels continuously, the bottom part of the object moves further,
before its values are read out in the detector. This is causing the squeezing
of the object (Fig. 9a and b), which can be corrected by a projective
transformation.
The camera works with 50 Hz, which means every 20 ms the recording of a new
image is triggered. In our camera, the read-out of all the ny=480 rows
takes about τreadout=15 ms. Therefore, the shift of the object
during the read-out from one row to the next one is
Δx=vxrτreadoutny.
Using this shifting value, a transformation matrix can be set up
D=100Δx10001.
The application of this transformation before deblurring the image corrects
the shearing effect, as can be seen in Fig. 9c. It is important to note
that this effect occurs always in the x direction. Since the object may
move any direction in the image, vx in Eq. (16) and v in Eq. (3) are
not necessarily identical – only in the case, if the relative motion between
camera and object takes place in the x direction as shown in Fig. 9. Otherwise,
vx is the x component of the speed v, showing in the x direction,
which means along the pixel rows of the IR image.
This squeezing effect can be also well observed in the images of Fig. 2. The
higher the speed is, the stronger the lower part of the image is shifted.
However, the small plate in each experiment was parallel to the y axis. The
restored images by applying first the shearing transformation according to
Eqs. (16) and (17), and then the motion deblurring, are shown in Fig. 10. The
images again become parallel to the y axis and the shape of the plates is
well restored.
Restored images of Fig. 2 moving a small plate with different speeds.
Infrared image of two plates moving with a speed of
1.5 m s-1(a);
after deblurring the objects (b), their shape and real temperature are well
visible.
Figure 11 demonstrates a further example for the advantage of this
reconstruction technique. Even if the different objects are hardly
distinguishable in the infrared image, after restoration they become well
visible. Figure 11 shows two plates moving with 1.5 m s-1 from the
left to the right. The plate on the right side has higher temperature than
the one on the left side. In the recorded infrared image, the two plates are
blurred and show almost the same temperature (Fig. 11a), but after
reconstruction their shape and temperature are correctly restored (Fig. 11b).
Infrared cameras can be well used to detect failures, e.g. surface cracks
in objects. In the so-called active thermographic inspection, first heat is
introduced to the object in a specific way, and due to the distribution and
the temporal change of the temperature the failures can be localized in the
infrared images (Maldague, 2001). One of the heating techniques is to apply
inductive heating, which is very efficient for ferromagnetic materials,
e.g. for steel. It can be excellently carried out in a scanning way, where
the objects are moved continuously below or through the induction coil and
the camera records the temperature distribution shortly afterwards
(Oswald-Tranta et al., 2010b; Oswald-Tranta and Sorger, 2012). Figure 12 shows a
small bell, which is moved below a coil, positioned on the left side, already
outside of the image. The inductive heating causes a high temperature
increase around a surface crack, which becomes very well visible and
detectable in the deblurred image. Also, in this case, the transformation with
the matrix of Eq. (17) has been used and the shape of the bell is very well
reconstructed.
A small, inductively heated bell with a motion speed 1 m s-1(a) and the deblurred image (b).
Blind deblurring
In all the previous examples, the exponential coefficient of the PSF and the
motion direction has been determined once for the measurement setup and this
has been used for restoring the images. This technique works in many cases,
e.g. in process control or in non-destructive testing, if the objects always
move with the same speed into the same direction.
Blurred temperature profile from Fig. 5a and the fitted exponential
function.
However, it is also possible to determine the PSF from the image itself, provided
there is a small hot point in the image somewhere. Theoretically, the PSF is
the answer to a Dirac delta perturbation, which means a small hot spot in the
ideal image becomes the shape of the PSF in the blurred image. Figure 5a
shows the deblurred image of the small moving ball. By extracting a profile in
the motion direction, one can fit an exponential function (see Fig. 13) to
it, and the obtained coefficient can be used as τim.
Infrared image of a small warm ball (a), gradient image with marked
motion direction (b), edges of the gradient image (c); restored image (d).
If the motion is linear but its direction is not known, this can be also
determined for many cases from the image itself (Oswald-Tranta et al., 2010a).
Figure 14 demonstrates this situation: in the first step, the gradient image
is calculated, in which the edges show the motion direction. After rotating
the image, the PSF can be determined by fitting an exponential function. This
technique works not only for such a simple case as a small ball in the image
but also for images where several spots have significantly higher temperature
than their surroundings.
The technique even works if the motion itself is not linear, but its path is
known. Using this a priori knowledge, an additional step is necessary to
transform the image first into a linear motion. For example, a rotation can be
transformed into a linear motion (Oswald-Tranta and Sorger, 2012), which can be
deblurred, and then by applying the inverse transformation an image with high
contrast is obtained. Figure 15 shows the infrared image of two small balls
with the same temperature, fixed to a stick which is rotating around one of
its ends. The circle of the ball on the right side has a larger radius and
thus a larger speed. Due to the larger blurring, its temperature seems to be
lower than the left ball's temperature. Figure 15 shows also the intermediate
steps for determining the rotation centre and the transformed image to a
linear motion. After the deblurring and the inverse transformation, both balls
have again a circular shape and their temperature is also restored to show
the same value.
Infrared image of two small balls circularly moving (a), gradient
image to determine the rotation centre (b), image transformed to linear
motion (c) and the restored image (d).
Even if the motion direction is not exactly a straight line or a precise
rotation, very good results can be achieved with an estimated direction and
speed. Figure 16a shows a hand moving in front of the camera. Its path is
arbitrary but almost goes along a straight line. With an approximated linear
motion, the image has been deblurred and its result is shown in Fig. 16b. The
image is sharp, even the ring on the finger becomes well visible.
Infrared image of a hand with motion blurring (a) and the
reconstructed image (b).
Summary and conclusion
It has been shown that in many cases the motion blurring in the image of a
microbolometer camera can be eliminated or strongly reduced. If the correct
PSF is used, a good estimation of the correct temperature can be calculated.
Several experimental examples have been presented and the main factors
influencing the temperature reconstruction have been identified.
If the speed and the direction of the motion are well known, then the PSF
needs to be determined once, and it can be used for deblurring of the
subsequent images. If the speed and the direction are not known, then the
blurred image of a small hot spot which is present in the image corresponds
to the PSF, which can be then determined with a good
approximation from the image itself.
This motion deblurring technique can be excellently used, e.g. in industrial
applications, where the objects are moving always with the same speed, like
on a conveyor belt or rotating always on the same trajectory. In these cases,
the reconstruction parameters can be determined once and used for
the deblurring of the images.
This method is also well applicable if there is only one object moving in
the field of view of the camera. Even if its speed or moving direction is
not known, the blind algorithm for reconstruction delivers usually very good
results.
If many objects with different speeds and different directions are moving in
front of the camera, and also their trajectories are crossing each other,
then the blurred images of the individual objects will also cross each
other, causing a problem for this reconstruction technique.
No data sets were used in this article.
The authors declare that they have no conflict of
interest.
This article is part of the special issue “Sensor/IRS2 2017”. It is a result of the AMA Conferences, Nuremberg, Germany, 30 May–1 June 2017. Edited by: Klaus-Peter Möllmann Reviewed
by: two anonymous referees
References
DIAS GmbH: Pyroview 640L, User manual, 2011.
FLIR: Titanium SC7500, User manual, 2010.
Gonzalez, R. C., Woods, R. E., and Eddins, S. L.: Digital Image Processing
Using MATLAB, Gatesmark Publishing, 2009.Maldague, X.: Infrared and Thermal Testing, Nondestructive Testing Handbook,
Vol. 3.ASNT, Columbus OH, available at: www.asnt.org, 2001.
Oswald-Tranta, B.: Automated thermographic non-destructive testing,
Habilitation, University of Leoben, Austria, 2012.Oswald-Tranta, B.: Motion deblurring of infrared images, AMA conferences, Nuremberg,
10.5162/irs2017/i3.1, 2017.Oswald-Tranta, B. and Sorger, M.: Scanning pulse phase thermography with line
heating, QIRT J., 9, 103–122, 10.1080/17686733.2012.714967, 2012.Oswald-Tranta, B., Sorger, M., and O'Leary, P.: Motion deblurring of infrared
images from a microbolometer camera, J. Infrared Phys. Technol., 53,
274–279, 10.1016/j.infrared.2010.04.003, 2010a.Oswald-Tranta, B., Sorger, M., and O'Leary, P.: Thermographic crack detection
and failure classification, J. Electron. Imaging, 19, 031204-1-7, 10.1117/1.3455991, 2010b.
Rajagopalan, A. N. and Chellappa, R.: Motion Deblurring: Algorithms and
Systems, Cambridge Unversity Press, Cambridge, UK, 2014.
Vollmer, M. and Möllmann, K.: Infrared thermal imaging, WILEY-VCH Verlag
GmbH & Co., Weinheim, Germany, 2010.