Laser-tracker-based reference measurement for geometric calibration of phase-measuring deflectometry with active display registration

Phase-measuring deflectometry (PMD) with active display registration (ADR) is a ray-optics-based technique for the shape measurement of specular surfaces. To obtain quantitative results, the relative position of the cameras of the PMD–ADR setup needs to be determined by geometric calibration. Geometric calibration can be performed by inserting a planar mirror into the setup that brings all camera fields of view to overlap on an active pattern display. The mirror is tilted to multiple positions and each time the cameras capture the displayed images, which yields sufficient data to obtain the relative camera positions and the positions of the mirror. In this article, we give a more detailed description of PMD–ADR and its calibration. We also implement a laser-tracker-based reference method to measure the mirror positions and use its result to expose systematic errors in the geometric calibration.

1 Introduction

Phase-measuring deflectometry (PMD) is a geometric-optics-based measurement technique for the robust noncontact full-field measurement of specular surfaces (Knauer et al., 2004). PMD measures the deflection of light on a surface under test (SUT) by finding a correspondence between the camera image plane and the surface of a display that is used as active pattern generator. Beside its application in surface inspection (Werling et al., 2009; Höfer et al., 2016) and deformation (Li et al., 2014) and figure measurement (Su et al., 2010), PMD is often used to measure the SUT shape (Zhang et al., 2021c). To obtain the shape, additional regularization information must be provided. Many methods of regularization have been proposed (Knauer et al., 2004; Petz, 2006; Li et al., 2012; Liu et al., 2017; Xu et al., 2019; Yue et al., 2020; Wang et al., 2021), which unfortunately often limit the setup geometry.

Additionally, the display surface deviates from often assumed planarity, which leads to shape measurement errors. Bartsch et al. (2019) determine display shape as part of geometric calibration, but it is difficult to obtain characterization of the full display surface. The display may also change due to thermal expansion or gravitational influence (Huang et al., 2015a, b). Petz et al. (2020) stabilize the display in their PMD setup and employ a pair of cameras to measure its shape. They obtain a remarkable accuracy of 1 µm over a 50 mm SUT diameter, even for strongly curved SUTs. Zhang et al. (2021a) use a camera pair and digital image correlation to actively register 3D points of the display and obtain regularization by moving the display to multiple positions. However, their setup is limited to the measurement of nearly flat samples.

As a method of regularization and a solution to the nonideal behavior of the display, Bartsch and Bergmann (2020) proposed PMD with active display registration (ADR), which directly registers the display surface 3D points, a method we recently implemented (Sperling and Bergmann, 2023). In this paper, we extend the theoretical and experimental description of PMD–ADR. As geometric calibration of the setup components is an important part of PMD (Huang et al., 2018), we also consider the impact of possible geometric calibration errors on the shape reconstruction. In PMD–ADR geometric calibration, the camera observes the display via reflection on a planar calibration mirror, from which geometric constraints about the setup can be derived. Besides the geometric constraints from the reflected rays, the mirror position may also be determined using reference markers applied to the mirror (Petz et al., 2020). In other contexts, a laser tracker has been used to measure the position of optical mirrors (Gallagher, 2003; Burge et al., 2007; Huang et al., 2015a), albeit at the danger of damaging the mirror (Huang et al., 2015b). In this paper, we implement a laser-tracker-based method to measure the mirror tilt during geometric calibration, without touching the mirror surface. The deviations of mirror positions determined by the tracker and determined by geometric calibration give rise to a discussion on their impact on shape reconstruction.

2 Measurement principle

Figure 1 shows a schematic of the PMD–ADR setup. It contains a PMD setup, in which a camera (1) observes a display via reflection on the SUT. Additionally, two cameras (2 and 3) directly observe the display. Each camera records images with sinusoidal intensity distribution shown on the display. Using phase-shifting techniques (Huang et al., 2018), one can calculate a mapping between camera pixels and the phase; the phase is an encoding of the display surface by 2D coordinates in units of display pixels. The rays of vision of pixels of cameras 2 and 3 with corresponding phase values can be intersected, to obtain a mapping from phase values to 3D points. We reduce noise of the 3D points by fitting a 2D polynomial to them that parameterizes the display shape. The mapping is then interpolated to assign a 3D point to each camera 1 pixel. To measure the shape of an SUT, the display is moved to several distances. This yields multiple 3D points for each camera 1 pixel, through which a line can be fitted. By intersecting the line with the corresponding camera 1 ray of vision, a 3D point of the SUT is obtained. To reduce noise, a normal vector is determined from the angular bisector of the fitted line and the ray of vision. Integration of all normal vectors yields the SUT shape (Huang et al., 2017).

Figure 1 Schematic of PMD–ADR measurement setup. To measure the SUT shape, the display is moved multiple distances. Using correspondences of phase values, multiple 3D points can be calculated and assigned to each camera 1 pixel. Intersecting the line fit through the points with the corresponding ray of vision yields a 3D point of the SUT.

Since the 3D points of the display surface are determined directly by cameras 2 and 3, PMD–ADR does not require calibration of the display position and shape. Therefore, only the cameras' rays of vision and their relative position need to be calibrated. We use the generic camera model and vision ray calibration (Bartsch et al., 2021) to calibrate the cameras. The generic camera model is suitable because in our case cameras 2 and 3 have wide-angle lenses, which might be inadequately modeled by many standard camera models (Bothe et al., 2010). Additionally, the relative position of cameras 2 and 3 need not be determined because they are calibrated in one common coordinate system. The remaining step is geometric calibration of the position of cameras 2 and 3 relative to camera 1.

2.1 Geometric calibration

To obtain the transformation T_2&3,1 from the coordinate system O_C2&3 of cameras 2 & 3 to the coordinate system O_C1 of camera 1, a planar mirror is inserted into the setup which reflects light from the display to camera 1. The planar mirror is tilted to multiple positions and each time a phase measurement is being recorded by all three cameras. Starting with a guess of T_2&3,1, we can calculate 3D points of the display surface in O_C1 coordinates corresponding to camera 1 pixels, by the methods explained in Sect. 2. On the other hand, we can trace the camera 1 rays of vision to the 3D display points via reflection on the mirror, if we guess the mirror positions. The deviation between the traced rays and the 3D points now serves as the cost function. Minimization of the cost function with respect to the mirror positions and T_2&3,1 yields the desired estimate of T_2&3,1.

The cost function is a sum of squares of the minimal distances between each point and the corresponding reflected ray. Only a fraction of the data of all camera 1 pixels is used in the numeric calculation to reduce the computational effort. T_2&3,1 is parameterized by a translation vector and three Euler angles. Each position of the planar mirror is parameterized by a “mirror vector” m: a plane is described by its minimal distance d from the origin and a unit vector e which is perpendicular to the plane and points towards it. By combining both variables into $\mathit{m}=d\cdot \mathit{e}$ (Hesch et al., 2008), we have a one-to-one correspondence between planes and 3D vectors, ignoring planes that intersect the origin. We minimize the cost function with the MATLAB function fminunc (The MathWorks Inc., 2023a). Another option would have been a least-squares solver that, however, leads to a large Jacobian. The mirror vectors enter the cost function in a symmetric way via dot products, which simplifies calculation of analytical derivatives. Our approach is very similar to the general external camera calibration problem with a mirror (Hesch et al., 2008; Takahashi et al., 2012; Xu et al., 2019).

2.2 Measuring mirror positions with a laser tracker

A laser tracker is a device that tracks a reference target and measures its 3D coordinates (Muralikrishnan et al., 2016). We use a spherically mounted retroreflector (SMR) as target and nests in which the SMR can be placed with reproducible position. The nests are glued to the rear of the calibration mirror hold. By placing the SMR in each of these nests, the tracker can obtain the coordinates of points that are rigidly connected to the mirror.

During geometric calibration, the tracker measures the coordinates of the nests for each mirror position. To obtain the mirror vectors, we fit a plane through the coordinates of the nests. Although the nests generally do not lie on a plane, a rigid transformation of the nest coordinates will lead to the same transformation of the fit plane coordinates. Since the nests and therefore the fit plane are rigidly connected to the mirror, there exists a coordinate system O_T such that, if we take the coordinates of the plane to be with respect to O_T, the plane will coincide with the actual mirror surface and not the nests. We call O_T the tracker coordinate system, although by its definition it will not coincide with the actual center of rotation of the tracker device. Therefore we can obtain the mirror vectors directly from the fit planes of the nests.

To compare the mirror positions obtained from the tracker with those from the geometric calibration, they need to be transformed to a common coordinate system. Figure 2 shows a schematic of how to transform mirror vectors from a coordinate system O to another O^′. m is the mirror vector with respect to O, and m^′ is the mirror vector with respect to O^′. Imagine for a moment that the orientations of O and O^′ are the same. Because of the collinearity of m and m^′ we have ${\mathit{m}}^{\prime }=\mathit{\alpha }\mathit{m}$ for some α. But since the end points of both m and m^′ lie on the plane, their difference vector is parallel to the plane and orthogonal to the mirror vector. Therefore $\left(t+\mathit{m}-{\mathit{m}}^{\prime }\right)\cdot \mathit{m}=\mathrm{0}$. By inserting αm for m^′, one obtains $\mathit{\alpha }=\mathrm{1}+\left(\mathit{t}\cdot \mathit{m}\right)/{\left|\mathit{m}\right|}^{\mathrm{2}}$ and hence ${\mathit{m}}^{\prime }=\left(\mathrm{1}+\left(\mathit{t}\cdot \mathit{m}\right)/{\left|\mathit{m}\right|}^{\mathrm{2}}\right)\mathit{m}$. If the vectors are given in coordinate systems with different orientations, i.e., m is given in O, m^′ and t are given in O^′, and R is the coordinate rotation matrix from O to O^′, we have to rotate m to O^′ orientation and obtain

$$\begin{array}{}\text{(1)}& {\mathit{m}}^{\prime }=\left(\mathrm{1}+{\displaystyle \frac{\left(\mathbf{R}\mathit{m}\right)\cdot \mathit{t}}{{\left|\mathit{m}\right|}^{\mathrm{2}}}}\right)\mathbf{R}\mathit{m}.\end{array}$$

Figure 2 Transformation of mirror vector from system O to system O^′. If both systems had the same coordinate axes orientation, m and m^′ would be parallel, and their position vector difference would lie in the mirror plane, i.e., $\left(\mathit{t}+\mathit{m}-{\mathit{m}}^{\prime }\right)\cdot \mathit{m}=\mathrm{0}$.

3 Experimental setup

Figure 3 shows a photograph of the experimental setup. Camera 1 is an AVT Manta G-145B 30 fps equipped with an Edmund Optics 59873 lens. The aperture is adjusted to $f/\mathrm{16}$. Camera 1 is focused on the SUT and observes the display via reflection on the SUT. Cameras 2 and 3 are AVT 1800 U-508m cameras with Computar M0528-MPW3 wide-angle lenses. Their aperture is adjusted to $f/\mathrm{12}$, and they are focused to infinity to record a sharp image over a large volume. The display is a LG 27UP850-W device, whose gamma curve has been linearized by prior calibration. The display can be conveniently moved by a robot manipulator.

Figure 3 Experimental setup. Camera 1 observes the display via reflection on the SUT. Cameras 2 and 3 observe the display directly.

As SUTs we use a planar mirror and two convex spherical mirrors with radii of curvature of approximately 200 and 100 mm. All samples have a diameter of 50 mm, of which the clear aperture of 45 mm diameter is used for evaluation. Each SUT is measured individually by placing it in the setup. To obtain data for shape reconstruction, the display is moved to five distances and each time a phase measurement is performed. For the 100 mm spherical mirror, multiple measurements are being recorded for each distance. This is necessary to cover the camera 1 field of view, which is enlarged by reflection on the high-curvature SUT.

Prior to SUT measurements, data for geometric calibration are gathered using the setup depicted in Fig. 4. Instead of the SUT, a $\mathit{\lambda }/\mathrm{20}$ at 633 nm planar calibration mirror is placed in the setup. The mirror is tilted to 17 different positions and each time a phase measurement is recorded. The rear of the mirror holder can be seen in Fig. 5. It features four nests, which are bearings with a cavity where the SMR locks magnetically. The nests are in the field of view of the laser tracker. The tracked 0.5 in. SMR can be placed successively in each of the nests to obtain coordinates that are rigidly connected to the mirror.

Figure 4 Calibration setup. Camera 1 observes the display via reflection on a large planar calibration mirror. Cameras 2 and 3 observe the display directly. The tracker records the coordinates of the SMR which can be placed at four nests at the rear of the mirror holder.

Figure 5 Rear of the calibration mirror holder with four nests and the SMR attached to one of the nests.

4 Results and discussion

4.1 Evaluation of tracker data

The determination of nest coordinates by the tracker might be affected by multiple error sources that do not originate from the tracker itself, such as thermal expansion of the calibration mirror hold or an imperfect snapping of the SMR to a nest due to dust or scratches. We repeat for each mirror position the measurement of the four nest coordinates four times to check repeatability. For all measurements and all nests, the deviation of coordinates of a nest from their mean is less than 5 µm. Using the Kabsch algorithm (Kabsch, 1976), we also aligned the averaged nest coordinates of different measurements to check for systematic changes of the nest's relative coordinates. This yields a maximum deviation of the nest coordinates of 4 µm, which shows sufficient consistency of the tracker data.

As described in Sect. 2.2, we fit planes to the four nest coordinates to obtain mirror vectors. As each of the four nests is measured four times, we can also estimate the mirror vector statistics by creating 4⁴=256 samples of four nests and fit a plane to each sample. For example, the correlation (Draper and Smith, 1998, p. 143) of the first mirror vector is

$$\begin{array}{}\text{(2)}& \mathit{\rho }\left({\mathit{m}}_{\mathrm{1}}\right)=\left(\begin{array}{ccc}\mathrm{1}& \mathrm{0.97}& \mathrm{0.84}\\ \mathrm{0.97}& \mathrm{1}& \mathrm{0.69}\\ \mathrm{0.84}& \mathrm{0.69}& \mathrm{1}\end{array}\right).\end{array}$$

This shows significant correlation between the components, which is reasonable since a slight change of only one of the components can severely increase the deviation of the plane from the nests. The other components then have to change as well to compensate and keep the plane deviation at the measured level. The correlation must be considered when we do a maximum likelihood estimation based on the mirror vectors.

The mirror vectors of the tracker are given in some coordinate frame O_T. To compare them with the mirror vectors from the geometric calibration, we optimize the transformation from O_C1 to O_T such that the mirror vectors align as well as possible. As the components of each tracker mirror vector are correlated, we use generalized nonlinear least squares (Seber and Wild, 1989, pp. 27–28) and minimize ${\sum }_j{\mathit{f}}_j^{\mathrm{T}}{\mathbf{C}}_j^{-\mathrm{1}}{\mathit{f}}_j$ so that each difference f_j of mirror vectors is weighted by the inverse of the covariance matrix C_j of the tracker mirror vector of the jth measurement. To solve the optimization problem, we use the MATLAB function lsqnonlin (The MathWorks Inc., 2023b). After alignment, the angle between mirror vectors from the tracker and from geometric calibration is on average 21±3 and 30 arcsec at maximum. This shows the presence of systematic errors in the geometric calibration, as the difference of mirror angles is well above uncertainty.

4.2 Shape reconstruction

We use the geometric calibration result to reconstruct the shape of measured SUTs. A performance criterion for shape reconstruction is the deviation of the reconstructed shape from reference. The reference radii of the spherical mirrors were determined at Physikalisch Technische Bundesanstalt (PTB), Braunschweig, Germany, to be (99.993±0.001) and (200.069±0.001) mm. For our evaluation, the uncertainty of the reference radii is negligible. After fitting a plane or sphere with the respective reference radius to the SUT shape, we obtain fit residuals as depicted in Fig. 6. Peak-to-valley (PV) values for the residuals are shown in Table 1.

Figure 6 Shape residuals. (a) Flat mirror, (b) convex mirror with radius 200 mm, and (c) convex mirror with radius 100 mm. (d, e, f) Corresponding shape residuals obtained with merged geometric calibration; see Sect. 4.3. The shape residuals increase with curvature and are of similar size for both calibrations. However, for nonzero curvature, their shape is different. It shows that the deviation from reference of the mirror positions in geometric calibration leads to considerable changes of reconstructed shape for SUTs with nonzero curvature.

Table 1 Shape residuals' peak-to-valley (PV) results for multiple SUTs. Right column: maximum difference of the residuals to the residuals obtained with the merged geometric calibration. For an explanation of merged geometric calibration see Sect. 4.3.

The residuals of the flat mirror are small. It is therefore not suitable to expose calibration errors, or the general shape reconstruction accuracy, of the PMD–ADR method. Still, the good reconstruction of the flat mirror shape indicates consistency of the recorded phase data with the camera calibration. This means that camera 1 is probably calibrated well, and cameras 2 and 3 are probably calibrated well in the center of their field of view, where the phase values were recorded that go into shape reconstruction. For nonzero curvature, the deviations are larger. It is of interest if the geometric calibration error that caused the deviation of mirror vectors can cause such shape deviations.

4.3 “Merged” geometric calibration

To investigate whether the observed deviation of mirror vectors during geometric calibration can cause the observed shape deviations, we also perform a merged geometric calibration, where the mirror poses determined with the laser tracker are incorporated into the geometric calibration procedure. The optimization of the mirror vectors is thereby replaced by the optimization of T_T,1 to have the mirror vectors available in O_C1 coordinates. The merged geometric calibration result for T_2&3,1 can then also be used for shape reconstruction.

Figure 6 shows the shape residuals of both calibrations for comparison. For every SUT, the residuals are different but of similar magnitude. The large shape deviations of merged geometric calibration are not unreasonable, despite the more accurate mirror vectors, because the underlying physical model of calibration did not change. For example, both calibration algorithms do not incorporate deviations of the calibration mirror from planarity, if that was the relevant source of error. Table 1 features the maximum differences of residuals of both calibrations. The flat mirror residuals are small and almost identical, which again shows that a flat mirror is not able to give insight into geometric calibration errors. The other differences are of similar size as the residuals themselves. Additionally, the increase in shape deviation with curvature is similar for both calibrations. The error source in geometric calibration that causes the mirror vectors to deviate from the reference may therefore very well cause the observed shape reconstruction deviations. This is a valuable result to us, since increased modeling effort can now be concentrated at the geometric calibration and prior calibration steps in order to improve measurement accuracy. Error sources that only affect the SUT measurement can for the moment be omitted, for example, the asymmetric point spread function that originates from the catadioptric system formed by the camera and the SUT (Zhang et al., 2021b). However, the above result does not exclude error sources that enter during SUT measurement and influence the shape reconstruction in the same order of magnitude.

Multiple sources of error are possible. Refraction at the display cover glass can cause deviations of the measured phase (Reh et al., 2014; Petz et al., 2020). We incorporated a cover glass correction algorithm and verified it using simulation, but for reasonably guessed cover glass parameters we were not able to reduce the deviation of mirror vectors or the shape deviations. We also excluded an imprecisely known display pixel pitch as error source, by comparing phase measurements obtained from the PMD–ADR display with phase measurements obtained from a display with precisely known pixel pitch, both recorded with the same cameras. Improving PMD–ADR accuracy will therefore be an ongoing topic in the future.

5 Conclusions

We have presented phase-measuring deflectometry (PMD) with active display registration (ADR), which is a shape measurement technique for specular surfaces. Geometric calibration is an important step for PMD–ADR. As one result of geometric calibration, one obtains the positions of the calibration mirror. We implemented a laser-tracker-based method to perform a reference measurement of the calibration mirror positions. The reference method yields different mirror positions than the geometric calibration. This shows systematic errors in the geometric calibration. By employing a merged geometric calibration, we were able to show that these errors are sufficient to yield shape reconstruction deviations of similar size as the observed deviations. This result helps us to guide our future efforts to improve modeling towards the calibration stage of PMD–ADR.