1 Introduction
Estimation of the deformation parameters of a target (either objects or texture) is a fundamental technique mainly for computer vision applications such as registration
(GayBellile et al., 2010; Cao et al., 2018; Balakrishnan et al., 2019) and tracking (Tan et al., 2014; Wang et al., 2019, 2020). If the geometric deformation model is constrained to only rotation and translation then the deformation is rigid. Affine or projective transformations can express more complex deformation, while practical applications, such as medical image analysis (Rueckert et al., 1999; Oliveira and Tavares, 2014) and morphing (Alexa, 2002; Scherhag et al., 2019), often involve nonrigid deformation with more degrees of freedom. Furthermore, the deformation between two images can be global, local, and even spaceinvariant, which makes the problem more challenging because the movement vector of each pixel is required to be estimated independently while preserving the smoothness. This technique can estimate the deformation parameters by deforming a template image such that the similarity between the deformed template and a target image is maximized. The procedure of similarity maximization can be cast as an optimization problem by treating deformation parameters as decision variables and the similarity as the objective function.
As to the geometric deformation model, freeform deformation (FFD) (Sederberg and Parry, 1986) models the deformation by manipulating control points arranged in a regular lattice over the target. Each pixel moves based on weights by basis functions and displacements of surrounding control points. Bspline basis functions are generally used to weigh displacements (Hsu et al., 1992). The larger the number of control points is, the more finely the deformation can be modeled. In FFD, the influence of a control point is limited to neighbor pixels, which brings benefits with respect to ability in modeling and computational cost (Crum et al., 2004). Due to these characteristics, FFD is allowed to model highly free and subtle deformation. However, optimization with FFD’s parameters is challenging since an optimizer needs to treat the displacements of all control points as decision variables, and it is obvious that the expressive power of the deformation model is proportional to the number of parameters. Moreover, because each control point can affect multiple regions, an improvement of similarity in one region may negatively make similarity in other regions worse, i.e., there exist conflicts between regions.
To alleviate the above issues, we introduce a new idea to estimate the FFD parameters by casting the deformation estimation problem as a multiobjective optimization problem (MOP), which can be effectively solved by multiobjective evolutionary algorithms (MOEAs). The overview of our algorithm is shown in Fig. 1
. A template is spatially divided into several groups and the similarity measure over each group is treated as a singleobjective function and independently computed. Each group consists of patches, and the pixels in each patch are affected by the same control points. A MOP requires simultaneous optimization of two or more objectives that conflict with each other. In our problem setting, we aim to find Pareto optimal solutions given none of the groups can be improved without degrading some of the other groups, which can be solved by certain offtheshelf MOEAs. In addition, we adopt a coarsetofine strategy using image pyramids to improve the estimation capability, especially for large deformations. Specifically, the optimization starts at the top of the pyramid (i.e., the lowest resolution image) and is executed at each level of the pyramid. The number of control points is gradually increased as the resolution increases, and the interpolation of control points is realized by mesh subdivision to allow finegrained deformation. Also, a postprocessing method is proposed to integrate Pareto optimal solutions into a single output as the decisionmaking procedure. For each group in Fig.
1, the groupwise deformation parameters with the highest groupwise similarity are adopted. These groupwise deformation parameters are aggregated into a final solution. We perform comparative experiments using both synthetic and realworld data to show the effectiveness and usefulness of our method. In conclusion, our contributions are threefold.
The deformation estimation problem is cast as a MOP by spatially dividing an image into multiple groups accompanied by independent similarity measures.

The estimation capability is improved by a coarsetofine strategy, which is realized by building image pyramids and conducting mesh subdivisions at each level.

A postprocessing method is proposed to integrate Pareto optimal solutions into a final output.
The rest of this paper is organized as follows. We present related works of deformation estimation and MOEAs in Sec. 2 followed by a brief review of FFD model in Sec. 3. The overview of three offtheshelf genetic algorithms (GAs) are given in Sec. 4. In Sec. 5, we describe the details of the spatial multiobjective problem and the coarsetofine optimization strategy. The experimental results are shown in Sec. 6. Conclusion is given in Sec. 7.
2 Related Work
2.1 Deformation estimation between two images
Many methods have been proposed to deal with deformable surfaces, which can be roughly categorized as featurebased methods and pixelbased methods. The former category estimates deformation parameters by using feature correspondences commonly extracted from two images. The latter category maximizes the similarity calculated using dense pixels directly. Also, hybrid methods have been studied to incorporate the advantages of both approaches (Zhu et al., 2009; Pizarro and Bartoli, 2012; Wu et al., 2013).
Featurebased methods (Pilet et al., 2008; Salzmann and Fua, 2011; Tran et al., 2012; Ngo et al., 2016; Wang et al., 2019)
estimated deformation parameters based on the correspondences between feature points between the template image and the target image. The accuracy largely depends on the quality of the correspondences. Therefore, the elimination of outliers from the extracted feature set is an essential process. However, the large number of parameters in freeform deformation makes it difficult to apply standard methods such as RANSAC
(Tran et al., 2012). Other limitations are: 1) In the case of featureless images, feature points are hard to be detected. Without inlier correspondences, the parameters cannot be appropriately estimated. Especially in the case of nonrigid transformations, more corresponding points are required (Bartoli and Zisserman, 2004). 2) Local features such as SIFT (Lowe, 2004) and ASIFT (Morel and Yu, 2009) are susceptible to complex transformation, which may largely degrade the confidence of correspondences when complex transformation occurs.The purpose of pixelbased methods (Bartoli and Zisserman, 2004; GayBellile et al., 2006; Malis, 2007; Hilsmann et al., 2010; GayBellile et al., 2010; Tan et al., 2014; Zhang and Akashi, 2015, 2016) is to solve the minimization problem of the cost function consisting of a data term and some restrictions such as smoothness term calculated from pixel intensities. The data term is usually defined as the sum of the intensity differences between the pixels of the template image and the corresponding pixels in the deformed target image. Such methods are less dependent on image features compared to featurebased methods. In addition, the capability of dealing with selfocclusion is a notable point. Since only a few features typically exist near the selfocclusion boundary, the pixelbased methods are more reasonable in such cases (GayBellile et al., 2010; Pizarro and Bartoli, 2012). In (GayBellile et al., 2010), a penalty term called shrinker is incorporated into the cost function. The shrinker term acts to shrink the displacement in order to make selfoccluded areas disappear. (Pizarro and Bartoli, 2012)
employed a pixelbased approach to refine the deformation parameters given by a proposed featurebased method. When selfocclusion or strong deformations are involved, the hybrid method shows better results than only using the featurebased method. There also exist researches to maximize the similarity under the framework of evolutionary computation with a single objective
(Zhang and Akashi, 2015, 2016). There also exist methods achieving the minimization of the cost function by employing nonlinear least squares solvers, such as the GaussNewton algorithm (Bartoli and Zisserman, 2004; GayBellile et al., 2010; Pizarro and Bartoli, 2012), the Levenberg–Marquardt algorithm (GayBellile et al., 2006; Hilsmann et al., 2010), and the learningbased methods (Tan et al., 2014). To the best of our knowledge, exploiting evolutionary algorithms (Klein et al., 2007) or multiobjective optimization approaches (Alderliesten et al., 2012; Pirpinia et al., 2019) to deal with deformable surfaces have been sparsely treated so far. Our previous work appearing in GECCO2019 addressed this problem by using a modified singleobjective GA (Nakane et al., 2019). Different from the previous work, in this paper, we attempt to adopt evolutionary algorithms for solving this problem by casting it as a multiobjective optimization problem.2.2 Multiobjective evolutionary algorithms (MOEAs)
EAs are optimization algorithms inspired by Darwin’s evolutionary theory, such as the GA, evolutionary strategy, and evolutionary programming. These algorithms share a common framework in which many candidate solutions are simultaneously dealt with and stochastic operations are iteratively applied. Because of the powerful exploration capability, EAs have been applied in a variety of computer vision tasks. Interested readers can also refer to the survey (Nakane et al., 2020). EAs are also effective tools for solving MOPs. The populationbased search procedure provides the advantage of finding the Pareto optimal solutions in a single run. MOEAs use dominance relation to rank solutions in an objective space consisting of conflicting objectives. In particular, representative MOEAs, such as nondominated sorting GAII (NSGAII) (Deb et al., 2002), strength Pareto EA2 (SPEA2) (Zitzler et al., 2002), and Pareto enveloped based selection algorithmII (PEASII) (Corne et al., 2001), include a mechanism that preserves nondominated solutions in every generation, called elitism, and hence these algorithms can outperform nonelitist MOEAs by preventing the loss of good solutions (Vachhani et al., 2015). Since the goal of MOEAs is to provide solutions that are widely distributed on the Pareto front, MOEAs are also required to maintain the diversity of solutions. In NSGAII, a crowding distance was proposed which is the sum of the distances between the two nearest solutions for each objective. SPEA2 used the inverse of the distance to the kth nearest solution as the density. PEASII divided the objective space into several hyperboxes and counted the number of solutions within them. The density was assigned to each hyperbox as the number of solutions contained. On the other hand, MOEAs are less effective for problems with four or more objectives, i.e., manyobjective optimization problems (MaOPs). The main reason is that as the number of objectives increases, the condition of dominance becomes more complex. More objectives lead to a greater proportion of nondominated solutions, and hence the ability of convergence toward the Pareto front decreases (GarzaFabre et al., 2009). There are several strategies to adapt MOEAs to MaOPs (Li et al., 2015a), such as dimensionality reduction (Saxena et al., 2013; Wang and Yao, 2016) and use of indicators (Bader and Zitzler, 2011; Li et al., 2016). Among them, one representative strategy is via decomposition, e.g., MOEA based on decomposition (MOEA/D) (Zhang and Li, 2007), where a MaOP was decomposed into singleobjective subproblems using weighting vectors, and referencepoint based manyobjective NSGAII (NSGAIII) (Deb and Jain, 2014), where an objective space was divided by reference vectors. There also exist various improved versions of MOEA/D (Xu et al., 2019; Zhang et al., 2020) and NSGAIII (Yuan et al., 2016; Cui et al., 2019). Combination of both merits is also shown in (Li et al., 2015b).
3 Deformation Model
Deformation estimation is achieved by registering the template image to the target image . In order to deform , we employ FFD combined with cubic Bsplines using control point meshes. For of pixels, control points are arranged on the lattice with horizontal spacing and vertical spacing (i.e., the outmost control points are outside the region of ), as illustrated in Fig. 2. Each control point is assigned a displacement vector representing the distance and direction from the initial position, and the movement of a certain coordinate on is determined by surrounding control points, which is defined as:
(1) 
where , and are the cubic Bspline basis functions (Lee et al., 1997). Then, the pixel coordinate on corresponding to is given by the transformation function :
(2) 
Since the transformation using Eq. 2 is a forward warping procedure and consists of real numbers, there is a problem that rounding operation is necessary to obtain pixel intensities. Such a process results in a large number of “holes” in the deformed template image, as illustrated in Fig. (a)a and Fig (b)b. An alternative is to employ backward warping that corresponding to is computed using the inverse transformation and interpolation scheme can be used to obtain the pixel intensity. According to (Schwarz, 2007), can be defined using an approximation:
(3) 
The deformation obtained in Eq. 3 and its difference from Eq. 2 are illustrated in Fig. (c)c and Fig. (d)d, respectively. In conclusion, Eq. 1 is employed for modeling geometric deformation and Eq. 3 is employed as the image transformation for registration in this paper.
4 Review of Genetic Algorithms (GAs)
To further explain how we use the multiobjective optimizer in Fig. 1 to solve the optimization problem, we provide a short review on GAs in this section. GA is one of the leading algorithms in natureinspired optimization methods. For a population consisting of a number of candidate solutions, the GA gradually optimizes the by iteratively applying genetic operators. One of the unique features of the GA is genotype representations for candidate solutions. Each candidate solution, called individual , is encoded into an internal representation, such as a bit string or a realvalued vector, in order to apply genetic operators. The genotype representations can allow genetic operators to adapt to different problems flexibly.
The genetic operators mainly consist of parents selection, crossover and mutation. The procedure of the simple GA (Holland, 1975) is briefly listed as follows:
 Step 1:

Set and generate individuals in randomly.
 Step 2:

Select individuals as parents from with parents selection operator.
 Step 3:

Generate an offspring population from parents by crossover and mutation operators.
 Step 4:

Evaluate all individuals in .
 Step 5:

Select individuals from as .
 Step 6:

Set and return to Step 2 until the termination criterion is satisfied.
Parent selection is typically implemented as probabilistic selection biased by evaluation values (i.e., individuals with better evaluation values are assigned higher probabilities.) The crossover operation generates offspring through the genetic recombination of multiple parents. The mutation operation randomly changes genes of offspring with low probability. Step 5 is also known as survivors selection, that is, all individuals in
and compete in order to become members of according to their evaluation values. The simple GA directly adopts as . Besides, the elitism strategy, which preserves the best individual in the pool is often adopted.NSGAII (Deb et al., 2002) is a representative GAbased algorithm for solving MOPs. The key idea of the NSGAII is to introduce the selection criterion using two sorting approaches. The first one, called fast nondominated sorting, iteratively extracts a nondominated set from the population and assigns a rank to each according to the order in which they are extracted. Another one, called crowding distance sorting, determines the priority in by the crowding distance which represents the density of neighboring individuals in the solution space. At last, individuals are selected from consisting of to , where . Insufficient individuals are taken from according to the crowding distance. Fast nondominated sorting promotes convergence to the Pareto front, while crowding distance sorting maintains diversity on the Pareto front. Moreover, elitism is ensured by using in survivors selection.
NSGAIII (Deb and Jain, 2014) is a variant of NSGAII which focuses on solving MaOPs. Instead of crowding distance sorting, reference lines connecting the origin with reference points evenly distributed on the evaluation value space are used to maintain the diversity of the population on the Pareto front. Each is associated with the closest reference line in the perpendicular distance. After is determined by fast nondominated sorting in survivors selection, the number of individuals in associated with each reference line, called niche count, is logged. NSGAIII then iteratively selects the individual in which is associated with the reference line of the lowest niche count. Reference points can relieve the algorithm in adaptively maintaining population diversity. In addition, users can obtain only a part of the Pareto front as required by manually distributing reference points.
5 Spatial MultiObjective Optimization
As described in Sec. 3, the deformation of the template image is determined by the displacement of each control point. Therefore, the purpose of the optimization procedure is to calculate the displacements where the deformed template image matches the target in the target image most. The principal contribution of this work is to cast this task as a MOP by spatially partitioning the template into groups. Each group is assigned a singleobjective function of the similarity measure. The overview of the optimization procedure is shown in Fig. 4. The procedure starts from building pyramids for both the template image and the target image, then the optimization is performed with the pyramids in a coarsetofine scheme. The key advantage of this framework is that the population optimized at each level can be inherited as the initial population of the next level. To ensure the consistency of parameter inheritance from lowresolution level to highresolution level, a subdivision method considering control point mesh is employed to achieve a natural interpolation of additional displacements, which allows the optimized population to be directly inherited.
Final candidate solutions are obtained by optimizing the population at the bottom of the pyramid (i.e., the image in the original resolution). The subsequent effort lies in how to determine a final solution as output from multiple solutions on the Pareto front. In addition to a direct selection approach, a postprocessing approach using multiple candidate solutions is also proposed.
5.1 Optimizing deformation parameters via MOEAs
Three GAs described in Sec. 4 are employed and compared to optimize the displacements in the experiment. Since each control point can move within the plane, the total number of decision variables is . To represent these variables as genes, we use realvalued coding because each displacement is a 2D vector of real values. For clarity, we denote the displacement of a single point by , and an individual is represented by the vector concatenating the displacements of all the points as follows,
(4) 
can directly represent a candidate control point mesh. The initial population is iteratively optimized by genetic operators. As to the evaluation of each , for simplicity, a singleobjective function is firstly introduced, which is combined with the simple GA and compared to multiobjective GAs in the experiment. In the objective function, mean absolute difference with respect to intensities is used as the similarity measure. As introduced in Sec. 3, we use backward warping to find the correspondences for the calculation of objective function, hence sampling is performed on the target image. Let and denote the entire region of the template image and the target image , respectively, and we can further define the sampling region as:
(5) 
The singleobjective function for the simple GA is given by:
(6) 
Based on Eq. 6, spatial multiobjective functions can be easily defined. We first define patch as a region consisting of pixels affected by the shared control points, i.e., consists of patches. The group partitioning is achieved by dividing all the patches into groups, where is the number of objective functions. We denote the region of th group as . One objective function is assigned over each group, which can be written by modifying Eq. 5 and Eq. 6:
(7) 
(8) 
where . Therefore, multiobjective GAs evaluate individuals based on the following vector function:
(9) 
Because each control point affects multiple patches, their similarity functions can “conflict” with each other, which is also considered in the case of groups consisting of multiple patches. This is an important motivation for adopting MOEAs because a singleobjective optimizer can hardly solve such a conflicting problem efficiently (Deb et al., 2002). The number of regions is adjustable as a hyperparameter (increasing makes optimization significantly more difficult). Pareto optimal solutions are supposed to obtain more appropriate solutions than optimizing a singleobjective.
5.2 Coarsetofine strategy
An iterative framework using image pyramids can be employed to alleviate the difficulty of deformation estimation with large displacements (Hilsmann et al., 2010; GayBellile et al., 2010). Estimation starts from the lowest resolution image for a rough estimation, and more accurate estimations are achieved as the image resolution increases. Specifically, we adopt Gaussian pyramid which iteratively generates low resolution images through Gaussian smoothing. The assignment of pyramid level indices follows the order in which estimation is performed (i.e., the first and the th level images are with the lowest and the highest resolution, respectively). Both the image width and height are halved and hence the th level image has resolution of the th level image.
With the increase of image resolution, it is also necessary to increase the resolution of the control point mesh to inherit deformation parameters between different levels. To this end, for a certain level, interpolating new control points without destroying the mesh configuration of the previous level is required. In this work, we adopt the Catmull–Clark subdivision (Catmull and Clark, 1978) for the mesh subdivision. The purpose of subdivision is to update the control point mesh with respect to each individual from the optimized to mesh, where and (i.e., and are fixed for all levels). The Catmull–Clark subdivision algorithm generates a subdivided mesh by inserting new control points and updating the existing control points. As illustrated in Fig. 5
, the points on the subdivided mesh can be classified into three types: face points, edge points, and vertex points. A face point is inserted to a patch. Assuming that the vertices of a patch are
and , the face point is computed as their centroid,(10) 
An edge point is inserted to the edge shared by two patches. Assuming that two face points are and , and two endpoints of the edge are and , the edge point is computed as follow:
(11) 
A vertex point is the updated point of a vertex shared by the four patches. Denoting that the average of the four face points is , and the average of the midpoints of the four edges which share as one of the endpoints is , the vertex point is computed as follows:
(12) 
Note that the edge points and vertex points outside the dashed rectangle are not calculated, as surrounding control points are needed for calculation. After interpolation of control points, all the displacements are doubled to fit the increase of resolution in the image pyramid. An example of the subdivision process is shown in Fig. 6. It can be observed that the subdivided mesh in Fig. (b)b can maintain the shape of the previous mesh in Fig. (a)a well. Therefore, this subdivision step is useful for providing a good initialization for optimizing the image of the next level in the pyramid.
5.3 Decision of the final output
We introduce a postprocessing procedure to decide the final single solution as the output from the optimized population. A natural idea is to define the best solution as the individual with the smallest sum of the objective function values (i.e., maximum similarity). However, in MOEAs, such an approach can lose most of the valuable information of the Pareto optimal solutions. We propose a postprocessed solution by exploiting these suboptimal solutions, as illustrated in Fig. 7. For each group, the solution with the smallest value of the corresponding objective function is extracted, which provides the control points that only affect the corresponding group. The final output is created by aggregating control points provided from all the groups. For shared control points, the average of their displacements is computed.
6 Experimental Results
The effectiveness and usefulness of solving the deformation estimation problem with the multiobjective scheme are verified using both synthetic data (Sec. 6.1) and realworld images (Sec. 6.2). We compared the following four settings: the simple GA with a singleobjective, NSGAII and NSGAIII with twoobjectives respectively, and NSGAIII with fourobjectives. In the twoobjective setting, patches are divided equally and vertically into two groups (e.g., for a lattice including patches, each group consists of patches). Similarly, patches are divided vertically and horizontally into four groups in the fourobjective setting (e.g., each group consists of patches for a lattice). The number of pyramid levels is fixed to three in all experiments. To reduce the computational cost, pixel sampling is performed for individual evaluation by scanning the target image at five pixel intervals. For the implementation of three GAs, the Platypus package^{1}^{1}1https://github.com/ProjectPlatypus/Platypus is used, which is an evolutionary computation framework in Python which includes many MOEAs. To ensure the correctness and fairness of the experiment, we only manually set several essential parameters and fix other parameters following the default setting throughout the experiment. Specifically, the number of evaluations is set to 10000 and the number of reference points of NSGAIII is set to 100 for the twoobjective setting and 120 for the fourobjective setting. For fair comparisons, all experiments are executed five times with different random seeds considering probabilistic operations. The initial population for all settings is kept the same with respect to each random seed.
6.1 Comparison with synthetic data
To only focus on verifying the estimation ability rather than robustness against noises, we prepare five images in pixels for generating template images and deformed target images, as shown in Fig. 8. The corresponding template images are obtained by cropping pixels regions from the center of each image. Eight types of deformations are used for generating a deformed target image, the parameters are:

Deformation (2 types): an image is deformed into a wavy shape by moving the control points according to a sine curve. In addition to verticalonly displacements, a combination of both vertical and horizontal displacements is used.

Lattice size (2 types): and lattices are used for the bottom image of the pyramid.

Ranges of decision variables (2 types): we use and as the range of each decision variable. Ranges are limited to make sure that control points do not overlap with each other spatially.
As a result, there are 40 (i.e., 5 images 8 types of parameter settings) types of deformed target images in total. These images are generated by using backward warping, and hence the ground truth of displacements can be obtained. In this section, we evaluate each result based on not only the root mean square error (RMSE) but also the mean Euclidean distance error (MEDE). RMSE is calculated from all the pixels between the deformed template image and the sampling region . MEDE is calculated based on all the ground truth displacements.
6.1.1 Results of vertical wavy images
Comprehensive numerical results of the best solutions with respect to the vertical wavy images are summarized in Table 1. For each combination of image setting and algorithm, the minimum, maximum, and average values based on five random trials are investigated. As can be observed by comparing the four algorithms in terms of RMSE and MEDE, it is clear that the twoobjective algorithms can achieve better results in most cases. Examples of visual results are shown in Fig 9, from which we can observe that these methods can estimate deformation parameters correctly for all the test images. By contrast, GA with singleobjective achieves the best result only once in terms of the average RMSE and zero times in terms of the average MEDE value. The accuracy of GA can degrade significantly, e.g., lattice with range as illustrated in Fig. (a)a. These observations verify the effectiveness of the multiobjective approaches. In addition, we can observe that fourobjective NSGAIII gets poorer results than twoobjective algorithms. Focusing on average values of both evaluation criteria, fourobjective NSGAIII outperforms others for zero times regarding the RMSE and only once regarding the MEDE.
RMSE  MEDE  

Image  Lattice size  Decision  GA  Two  Two  Four  GA  Two  Two  Four 
variable  objective  objective  objective  objective  objective  objective  
range  NSGAII  NSGAIII  NSGAIII  NSGAII  NSGAIII  NSGAIII  
Plant  2.28E+00  2.01E+00  2.16E+00  2.59E+00  1.22E01  1.01E01  1.02E01  1.23E01  
3.00E+00  2.76E+00  2.53E+00  3.36E+00  1.51E01  1.17E01  1.24E01  1.83E01  
2.64E+00  2.30E+00  2.34E+00  2.93E+00  1.35E01  1.10E01  1.13E01  1.43E01  
5.01E+00  4.30E+00  4.72E+00  5.05E+00  2.65E01  2.13E01  2.55E01  2.76E01  
6.06E+00  5.55E+00  6.15E+00  6.90E+00  3.27E01  3.02E01  3.02E01  3.52E01  
5.56E+00  4.92E+00  5.16E+00  5.93E+00  2.94E01  2.62E01  2.68E01  3.11E01  
3.26E+00  2.73E+00  2.44E+00  3.26E+00  1.62E01  1.48E01  1.14E01  1.66E01  
4.50E+00  4.23E+00  3.78E+00  4.16E+00  2.31E01  2.09E01  1.95E01  2.14E01  
3.85E+00  3.34E+00  3.16E+00  3.65E+00  1.96E01  1.79E01  1.63E01  1.92E01  
6.12E+00  5.10E+00  5.16E+00  6.53E+00  3.35E01  2.82E01  3.42E01  3.66E01  
9.34E+00  9.68E+00  7.65E+00  8.51E+00  6.91E01  7.22E01  5.21E01  5.32E01  
7.37E+00  6.89E+00  6.84E+00  7.43E+00  4.67E01  4.53E01  4.41E01  4.64E01  
Sea  3.79E+00  3.85E+00  3.49E+00  4.13E+00  1.03E01  1.07E01  9.17E02  1.41E01  
4.19E+00  4.16E+00  4.36E+00  4.93E+00  1.28E01  1.20E01  1.24E01  1.57E01  
3.97E+00  4.04E+00  3.94E+00  4.50E+00  1.18E01  1.15E01  1.13E01  1.48E01  
8.08E+00  7.68E+00  8.05E+00  9.10E+00  2.22E01  2.26E01  2.46E01  2.80E01  
1.06E+01  9.11E+00  1.02E+01  1.04E+01  3.37E01  2.58E01  3.47E01  3.73E01  
9.10E+00  8.46E+00  9.11E+00  9.57E+00  2.66E01  2.41E01  3.00E01  3.16E01  
4.96E+00  4.19E+00  3.84E+00  4.71E+00  1.47E01  1.09E01  1.10E01  1.47E01  
7.10E+00  7.57E+00  6.02E+00  6.33E+00  1.94E01  2.41E01  2.01E01  2.14E01  
5.67E+00  5.37E+00  4.78E+00  5.35E+00  1.72E01  1.59E01  1.45E01  1.64E01  
8.90E+00  7.39E+00  6.72E+00  8.48E+00  3.85E01  2.68E01  2.61E01  2.89E01  
1.36E+01  9.32E+00  1.08E+01  1.15E+01  9.89E01  4.07E01  5.06E01  6.42E01  
1.14E+01  8.12E+00  9.03E+00  9.58E+00  6.49E01  3.12E01  3.66E01  3.97E01  
Rag  3.52E+00  3.89E+00  3.06E+00  3.96E+00  9.64E02  9.47E02  8.41E02  1.03E01  
4.65E+00  4.93E+00  5.01E+00  5.18E+00  1.22E01  1.16E01  1.21E01  1.32E01  
4.24E+00  4.17E+00  3.94E+00  4.64E+00  1.10E01  1.03E01  9.90E02  1.20E01  
8.12E+00  8.18E+00  8.24E+00  8.78E+00  2.17E01  2.10E01  2.15E01  2.27E01  
1.01E+01  9.24E+00  8.75E+00  1.04E+01  2.69E01  2.56E01  2.24E01  2.78E01  
9.21E+00  8.73E+00  8.54E+00  9.59E+00  2.51E01  2.28E01  2.20E01  2.61E01  
4.97E+00  4.08E+00  4.88E+00  5.25E+00  1.35E01  1.03E01  1.25E01  1.39E01  
7.35E+00  5.65E+00  6.21E+00  6.13E+00  2.23E01  1.51E01  1.69E01  1.54E01  
6.01E+00  4.93E+00  5.41E+00  5.56E+00  1.68E01  1.23E01  1.43E01  1.44E01  
9.58E+00  8.43E+00  7.21E+00  1.03E+01  2.61E01  2.31E01  2.03E01  2.81E01  
2.15E+01  1.32E+01  1.87E+01  1.17E+01  8.39E01  3.79E01  5.93E01  3.54E01  
1.72E+01  9.74E+00  1.07E+01  1.10E+01  5.99E01  2.75E01  3.07E01  3.19E01  
Alphabet  3.81E+00  3.43E+00  3.84E+00  4.55E+00  1.08E01  9.81E02  1.08E01  1.19E01  
4.69E+00  5.14E+00  4.45E+00  5.26E+00  1.41E01  1.42E01  1.23E01  1.67E01  
4.14E+00  4.15E+00  4.21E+00  4.90E+00  1.21E01  1.23E01  1.14E01  1.44E01  
7.46E+00  7.44E+00  7.67E+00  8.83E+00  2.04E01  2.07E01  2.36E01  2.76E01  
9.04E+00  9.55E+00  8.67E+00  1.10E+01  2.72E01  2.47E01  2.76E01  3.45E01  
8.44E+00  8.42E+00  8.33E+00  9.93E+00  2.36E01  2.29E01  2.52E01  3.05E01  
4.03E+00  3.98E+00  3.96E+00  5.17E+00  1.50E01  1.36E01  1.34E01  1.95E01  
5.93E+00  5.17E+00  5.21E+00  7.07E+00  2.42E01  1.74E01  1.71E01  2.33E01  
4.86E+00  4.76E+00  4.35E+00  6.07E+00  1.89E01  1.55E01  1.43E01  2.16E01  
1.14E+01  8.62E+00  7.73E+00  9.92E+00  3.99E01  3.48E01  2.15E01  3.71E01  
1.29E+01  1.12E+01  1.15E+01  1.28E+01  1.01E+00  4.63E01  4.32E01  4.92E01  
1.19E+01  9.50E+00  9.86E+00  1.16E+01  5.77E01  3.94E01  3.73E01  4.34E01  
Toad  1.56E+00  1.67E+00  1.22E+00  2.17E+00  1.06E01  1.12E01  1.23E01  1.43E01  
2.58E+00  2.40E+00  2.32E+00  2.35E+00  1.50E01  1.46E01  1.56E01  1.61E01  
2.03E+00  1.95E+00  1.92E+00  2.26E+00  1.31E01  1.25E01  1.35E01  1.51E01  
3.24E+00  2.76E+00  2.62E+00  3.53E+00  2.85E01  2.73E01  2.71E01  2.85E01  
4.97E+00  3.78E+00  3.47E+00  4.79E+00  3.92E01  3.36E01  3.41E01  3.47E01  
4.05E+00  3.27E+00  3.12E+00  4.25E+00  3.28E01  3.05E01  3.14E01  3.18E01  
2.19E+00  1.90E+00  1.56E+00  1.99E+00  2.31E01  2.26E01  2.20E01  2.02E01  
2.58E+00  2.51E+00  2.60E+00  3.45E+00  3.58E01  2.95E01  3.13E01  2.90E01  
2.39E+00  2.14E+00  2.10E+00  2.67E+00  3.21E01  2.68E01  2.52E01  2.56E01  
3.06E+00  2.97E+00  2.81E+00  3.44E+00  5.40E01  4.22E01  4.57E01  4.35E01  
4.53E+00  3.57E+00  4.91E+00  4.98E+00  8.54E01  7.35E01  7.92E01  5.01E01  
3.87E+00  3.23E+00  3.73E+00  4.08E+00  6.72E01  5.18E01  5.77E01  4.74E01 
The final solutions after postprocessing on Pareto optimal solutions obtained by multiobjective algorithms are compared in Table 2. As can be observed, the postprocessing successfully improves the estimation accuracy in many cases comparing to Table 1. In particular, fourobjective NSGAIII benefits most from the postprocessing. Focusing on the number of improved results on the average value, fourobjective NSGAIII performs the best 17 times on RMSE and 19 times on MEDE, while twoobjective NSGAII performs the best for 15 and 14 times, respectively.
RMSE  MEDE  

Image  Lattice size  Decision  Two  Two  Four  Two  Two  Four 
variable  objective  objective  objective  objective  objective  objective  
range  NSGAII  NSGAIII  NSGAIII  NSGAII  NSGAIII  NSGAIII  
Plant  1.98E+00  2.40E+00  2.37E+00  9.98E02  1.08E01  1.16E01  
3.04E+00  2.85E+00  3.40E+00  1.31E01  1.38E01  1.75E01  
2.45E+00  2.54E+00  2.80E+00  1.15E01  1.20E01  1.37E01  
4.50E+00  4.29E+00  4.98E+00  2.28E01  2.23E01  2.59E01  
5.61E+00  6.71E+00  6.81E+00  2.99E01  3.42E01  3.45E01  
5.12E+00  5.15E+00  5.70E+00  2.67E01  2.70E01  3.01E01  
2.55E+00  2.25E+00  3.07E+00  1.37E01  1.04E01  1.58E01  
4.27E+00  3.51E+00  3.79E+00  2.16E01  1.85E01  1.96E01  
3.23E+00  3.03E+00  3.38E+00  1.76E01  1.57E01  1.74E01  
5.21E+00  5.27E+00  6.04E+00  2.86E01  3.48E01  3.25E01  
9.51E+00  7.49E+00  8.34E+00  7.39E01  5.27E01  5.03E01  
6.73E+00  6.61E+00  6.87E+00  4.53E01  4.29E01  4.26E01  
Sea  3.36E+00  3.19E+00  4.11E+00  9.53E02  1.00E01  1.26E01  
3.78E+00  4.18E+00  5.00E+00  1.23E01  1.21E01  1.54E01  
3.61E+00  3.71E+00  4.53E+00  1.11E01  1.10E01  1.41E01  
7.73E+00  8.15E+00  8.45E+00  2.34E01  2.43E01  2.55E01  
9.29E+00  9.69E+00  1.11E+01  2.67E01  3.40E01  3.75E01  
8.32E+00  8.94E+00  9.45E+00  2.46E01  3.04E01  3.09E01  
4.00E+00  3.69E+00  4.46E+00  1.09E01  1.03E01  1.27E01  
7.43E+00  5.68E+00  5.59E+00  2.34E01  1.86E01  1.85E01  
5.21E+00  4.58E+00  4.92E+00  1.56E01  1.39E01  1.43E01  
6.72E+00  6.38E+00  7.84E+00  2.45E01  2.42E01  2.55E01  
9.33E+00  1.07E+01  1.11E+01  4.05E01  5.03E01  5.94E01  
7.83E+00  8.90E+00  9.12E+00  3.00E01  3.54E01  3.60E01  
Rag  3.90E+00  3.17E+00  4.08E+00  9.39E02  8.83E02  1.03E01  
5.02E+00  4.68E+00  5.18E+00  1.29E01  1.16E01  1.44E01  
4.25E+00  3.77E+00  4.67E+00  1.08E01  9.79E02  1.21E01  
7.59E+00  8.27E+00  8.78E+00  2.08E01  2.17E01  2.33E01  
9.32E+00  9.16E+00  9.85E+00  2.58E01  2.38E01  2.57E01  
8.44E+00  8.65E+00  9.45E+00  2.22E01  2.27E01  2.49E01  
4.04E+00  4.86E+00  4.72E+00  1.02E01  1.21E01  1.24E01  
5.47E+00  5.95E+00  5.82E+00  1.43E01  1.62E01  1.41E01  
4.70E+00  5.33E+00  5.23E+00  1.18E01  1.38E01  1.31E01  
8.14E+00  6.86E+00  9.25E+00  2.33E01  1.95E01  2.52E01  
1.35E+01  1.86E+01  1.14E+01  3.93E01  5.86E01  3.42E01  
9.66E+00  1.03E+01  1.02E+01  2.73E01  2.99E01  2.89E01  
Alphabet  3.58E+00  3.57E+00  4.30E+00  1.08E01  1.07E01  1.18E01  
4.86E+00  4.49E+00  5.02E+00  1.34E01  1.29E01  1.55E01  
4.11E+00  4.14E+00  4.72E+00  1.21E01  1.20E01  1.40E01  
7.28E+00  7.85E+00  8.84E+00  1.93E01  2.35E01  2.57E01  
9.51E+00  8.62E+00  1.02E+01  2.73E01  2.90E01  3.24E01  
8.10E+00  8.22E+00  9.58E+00  2.31E01  2.54E01  2.91E01  
4.00E+00  3.86E+00  5.01E+00  1.34E01  1.31E01  1.79E01  
5.04E+00  4.95E+00  6.51E+00  1.81E01  1.61E01  2.04E01  
4.64E+00  4.26E+00  5.73E+00  1.52E01  1.39E01  1.93E01  
8.45E+00  7.47E+00  9.00E+00  3.51E01  2.14E01  3.46E01  
1.10E+01  1.13E+01  1.20E+01  4.50E01  4.22E01  4.66E01  
9.56E+00  9.68E+00  1.10E+01  3.88E01  3.66E01  4.02E01  
Toad  1.55E+00  1.21E+00  1.91E+00  1.01E01  1.11E01  1.25E01  
2.52E+00  2.39E+00  2.58E+00  1.38E01  1.49E01  1.55E01  
1.93E+00  1.91E+00  2.24E+00  1.21E01  1.31E01  1.39E01  
2.99E+00  2.44E+00  3.78E+00  2.56E01  2.79E01  2.54E01  
4.29E+00  3.62E+00  5.14E+00  3.35E01  3.52E01  3.52E01  
3.68E+00  3.12E+00  4.64E+00  2.91E01  3.14E01  3.03E01  
1.86E+00  1.51E+00  1.90E+00  2.33E01  2.08E01  1.76E01  
2.50E+00  2.64E+00  3.40E+00  2.88E01  3.04E01  2.80E01  
2.12E+00  2.13E+00  2.60E+00  2.65E01  2.45E01  2.36E01  
2.92E+00  2.79E+00  3.18E+00  4.26E01  4.71E01  4.05E01  
3.67E+00  4.72E+00  4.81E+00  7.59E01  7.87E01  4.80E01  
3.20E+00  3.64E+00  3.94E+00  5.27E01  5.70E01  4.45E01 
6.1.2 Results of vertical and horizontal wavy images
The results of the best solutions with respect to the vertical and horizontal wavy images are shown in Table 3. Despite more complex deformations, the multiobjective approaches can still achieve good estimation. Several qualitative results are shown in Fig. 11. Comparing with Table 1, we can observe that the best results are irregularly distributed in terms of both evaluation criteria. Nevertheless, twoobjective NSGAII and NSGAIII are still better choices overall. In the case of lattice, GA can achieve the best result when the search space is small (e.g., the sea image). Fourobjective NSGAIII shows good results in the case of lattice setting, which achieves the best average MEDE for six times out of ten times. Although the number of groups is a hyperparameter to be handled carefully as mentioned in Sec. 6.1.1, our results show the trend that larger number of groups is more effective when dealing with complex and subtle deformations.
RMSE  MEDE  

Image  Lattice size  Decision  GA  Two  Two  Four  GA  Two  Two  Four 
variable  objective  objective  objective  objective  objective  objective  
range  NSGAII  NSGAIII  NSGAIII  NSGAII  NSGAIII  NSGAIII  
Plant  5.13E+00  4.38E+00  4.49E+00  4.95E+00  2.29E01  1.89E01  2.06E01  2.12E01  
5.58E+00  6.89E+00  5.65E+00  7.17E+00  2.64E01  3.45E01  2.55E01  3.27E01  
5.39E+00  5.45E+00  5.01E+00  5.80E+00  2.44E01  2.44E01  2.30E01  2.66E01  
9.42E+00  9.56E+00  9.51E+00  1.11E+01  4.94E01  5.27E01  4.86E01  6.19E01  
1.05E+01  1.08E+01  1.16E+01  1.25E+01  6.06E01  5.94E01  6.96E01  6.90E01  
1.01E+01  1.01E+01  1.05E+01  1.16E+01  5.53E01  5.66E01  5.71E01  6.57E01  
8.40E+00  6.10E+00  6.28E+00  5.13E+00  5.13E01  3.32E01  3.57E01  2.55E01  
9.95E+00  8.99E+00  8.92E+00  8.79E+00  5.76E01  5.43E01  5.18E01  5.27E01  
8.95E+00  7.82E+00  7.50E+00  7.37E+00  5.41E01  4.48E01  4.36E01  4.14E01  
1.02E+01  8.32E+00  9.84E+00  9.94E+00  6.07E01  5.18E01  6.63E01  6.22E01  
2.28E+01  1.22E+01  1.34E+01  1.17E+01  3.00E+00  7.95E01  9.62E01  7.07E01  
1.36E+01  9.97E+00  1.14E+01  1.05E+01  1.20E+00  6.44E01  7.45E01  6.80E01  
Sea  5.32E+00  5.41E+00  5.65E+00  7.00E+00  1.60E01  1.76E01  1.62E01  2.06E01  
6.29E+00  6.78E+00  6.40E+00  8.50E+00  1.98E01  2.21E01  2.15E01  3.12E01  
5.78E+00  6.15E+00  6.07E+00  7.54E+00  1.72E01  1.96E01  1.97E01  2.55E01  
1.09E+01  1.14E+01  1.23E+01  1.26E+01  4.94E01  4.90E01  5.35E01  5.90E01  
1.36E+01  1.32E+01  1.46E+01  1.45E+01  6.47E01  5.77E01  9.34E01  8.52E01  
1.25E+01  1.21E+01  1.34E+01  1.32E+01  5.78E01  5.28E01  6.60E01  6.60E01  
6.38E+00  6.85E+00  5.99E+00  7.31E+00  2.73E01  2.81E01  2.43E01  2.70E01  
9.62E+00  9.61E+00  9.90E+00  1.00E+01  4.97E01  4.65E01  5.17E01  4.62E01  
8.07E+00  8.41E+00  8.82E+00  8.47E+00  3.73E01  3.90E01  4.33E01  3.43E01  
1.17E+01  1.22E+01  1.29E+01  1.31E+01  1.02E+00  6.86E01  9.20E01  8.90E01  
1.74E+01  1.68E+01  1.52E+01  1.59E+01  1.56E+00  1.94E+00  1.57E+00  1.78E+00  
1.48E+01  1.48E+01  1.37E+01  1.45E+01  1.31E+00  1.39E+00  1.14E+00  1.29E+00  
Rag  6.54E+00  6.68E+00  7.07E+00  7.23E+00  1.74E01  1.59E01  1.70E01  2.02E01  
1.09E+01  8.72E+00  1.04E+01  1.08E+01  3.16E01  2.18E01  3.08E01  3.01E01  
8.38E+00  7.42E+00  8.27E+00  9.08E+00  2.26E01  1.88E01  2.15E01  2.48E01  
1.30E+01  1.31E+01  1.29E+01  1.61E+01  3.62E01  3.68E01  3.84E01  4.70E01  
1.57E+01  1.64E+01  1.53E+01  1.85E+01  5.19E01  5.22E01  4.93E01  6.77E01  
1.43E+01  1.45E+01  1.45E+01  1.76E+01  4.37E01  4.36E01  4.41E01  5.68E01  
1.33E+01  1.15E+01  1.29E+01  1.37E+01  4.61E01  3.73E01  4.54E01  5.07E01  
1.74E+01  1.50E+01  1.58E+01  1.63E+01  7.63E01  5.45E01  5.89E01  6.04E01  
1.53E+01  1.40E+01  1.47E+01  1.49E+01  5.90E01  4.87E01  5.27E01  5.43E01  
1.66E+01  1.71E+01  1.55E+01  1.89E+01  6.91E01  6.42E01  5.06E01  6.77E01  
2.66E+01  3.00E+01  2.86E+01  2.27E+01  2.02E+00  2.19E+00  2.22E+00  9.76E01  
2.29E+01  2.18E+01  2.22E+01  2.02E+01  1.35E+00  1.11E+00  1.23E+00  7.90E01  
Alphabet  5.15E+00  5.20E+00  4.78E+00  6.62E+00  2.03E01  2.13E01  1.95E01  2.67E01  
6.23E+00  5.70E+00  5.90E+00  8.25E+00  2.48E01  2.33E01  2.45E01  3.30E01  
5.51E+00  5.50E+00  5.35E+00  7.10E+00  2.21E01  2.21E01  2.24E01  2.88E01  
9.87E+00  1.09E+01  1.06E+01  1.25E+01  3.92E01  4.24E01  4.19E01  5.24E01  
1.18E+01  1.20E+01  1.29E+01  1.39E+01  5.08E01  4.85E01  5.58E01  6.40E01  
1.09E+01  1.13E+01  1.17E+01  1.32E+01  4.65E01  4.67E01  4.96E01  5.72E01  
6.93E+00  5.76E+00  4.34E+00  6.92E+00  3.21E01  2.49E01  1.93E01  3.15E01  
8.88E+00  6.93E+00  1.08E+01  8.43E+00  3.82E01  3.18E01  4.62E01  3.96E01  
7.52E+00  6.59E+00  6.82E+00  7.65E+00  3.49E01  2.95E01  3.00E01  3.60E01  
1.84E+01  1.19E+01  1.08E+01  1.06E+01  1.02E+00  5.98E01  5.37E01  5.77E01  
2.59E+01  2.26E+01  2.37E+01  1.48E+01  1.70E+00  1.44E+00  1.62E+00  7.67E01  
2.20E+01  1.68E+01  1.63E+01  1.26E+01  1.29E+00  9.69E01  9.61E01  6.60E01  
Toad  3.09E+00  3.05E+00  2.91E+00  3.13E+00  2.21E01  2.00E01  1.82E01  2.29E01  
3.88E+00  3.79E+00  3.42E+00  4.33E+00  3.52E01  4.26E01  2.57E01  3.86E01  
3.62E+00  3.40E+00  3.05E+00  3.83E+00  2.72E01  2.53E01  2.41E01  3.11E01  
6.72E+00  5.80E+00  4.63E+00  7.48E+00  5.08E01  4.70E01  5.02E01  5.93E01  
7.95E+00  7.14E+00  8.59E+00  9.89E+00  7.05E01  6.39E01  5.83E01  7.51E01  
7.48E+00  6.48E+00  6.16E+00  8.43E+00  6.13E01  5.50E01  5.46E01  6.60E01  
3.30E+00  2.67E+00  2.51E+00  3.36E+00  4.48E01  3.18E01  3.70E01  4.17E01  
3.91E+00  3.64E+00  3.73E+00  4.61E+00  7.05E01  6.74E01  7.19E01  5.13E01  
3.54E+00  3.08E+00  3.21E+00  3.95E+00  6.02E01  5.14E01  4.99E01  4.61E01  
7.57E+00  6.26E+00  5.95E+00  8.72E+00  1.35E+00  8.29E01  8.65E01  8.24E01  
9.98E+00  8.55E+00  7.39E+00  1.07E+01  1.92E+00  1.39E+00  1.33E+00  1.03E+00  
8.70E+00  7.62E+00  6.56E+00  9.58E+00  1.57E+00  1.20E+00  1.13E+00  9.22E01 
6.2 Comparison on realworld images
The usefulness of the proposed method under realworld scenarios is verified in this section. We use three different pairs of template and target images as shown in Fig. 12.

Texture (Fig. (a)a): the images capture a part of the undeformed/deformed texture printed on a piece of paper. The target image has two vertical bumps. We set the lattice as and the decision variable range as .

Sign (Fig. (b)b): the images capture a sign with undeformed/deformed text printed on a piece of wrapping paper. The lattice and decision variable range are set to lattice and , respectively.

Face (Fig. (c)c): the template image and the target image show a frontal face with serious expression and smile, respectively. The goal is to obtain deformation parameters that express smiling. We use the lattice and the decision variable range.
The sizes of the template images and target images are set the same as Sec. 6.1. Fig. (a)a and Fig. (b)b are captured by a web camera and Fig. (c)c is extracted from the FEI face database^{2}^{2}2https://fei.edu.br/~cet/facedatabase.html (Thomaz and Giraldi, 2010) and cropped. Because the ground truth of the real deformations is unknown, we evaluate results only using RMSE.
The results of best solutions and postprocessed solutions are shown in Table 4. These results show that multiobjective approaches can outperform the singleobjective approach in realworld situations. However, postprocessing fails to improve estimation accuracy in a number of cases (e.g., the face image). The estimation results for each image are shown in Fig. 13. It can be observed that the deformed texture image contains some highlight areas and the eye area of the face image is also shadowed. Hence, the deformation results for these corresponding areas show worse accuracy than the other areas. The groups including these areas cannot contribute to the postprocessing. As a limitation, the proposed method suffers from illumination changes due to the characteristics of the similarity measure.
Best solution  Postprocessed solution  
Image  Lattice size  Decision  GA  Two  Two  Four  Two  Two  Four 
variable  objective  objective  objective  objective  objective  objective  
range  NSGAII  NSGAIII  NSGAIII  NSGAII  NSGAIII  NSGAIII  
Texture  1.23E+01  1.21E+01  1.24E+01  1.18E+01  1.23E+01  1.23E+01  1.19E+01  
1.29E+01  1.25E+01  1.28E+01  1.25E+01  1.25E+01  1.28E+01  1.25E+01  
1.25E+01  1.23E+01  1.25E+01  1.21E+01  1.24E+01  1.26E+01  1.22E+01  
Sign  2.04E+01  1.95E+01  1.95E+01  2.04E+01  1.98E+01  1.95E+01  2.02E+01  
2.13E+01  2.03E+01  2.06E+01  2.07E+01  2.04E+01  2.10E+01  2.06E+01  
2.09E+01  2.00E+01  2.00E+01  2.05E+01  2.01E+01  2.00E+01  2.05E+01  
Face  7.55E+00  7.42E+00  7.37E+00  7.50E+00  7.47E+00  7.42E+00  7.50E+00  
8.01E+00  7.71E+00  7.61E+00  7.87E+00  7.88E+00  7.82E+00  8.08E+00  
7.72E+00  7.58E+00  7.48E+00  7.69E+00  7.65E+00  7.56E+00  7.77E+00 
7 Conclusion
In this paper, we proposed a novel deformation estimation method using MOEAs to tackle the conflicts based on the fact that each control point of the deformation model affects a local region rather than a single pixel. Our method casts deformation estimation as a MOP by dividing a template image into several groups consisting of patches with groupwise similarity defined as groupwise objective functions, which can be solved by offtheshelf MOEAs. To handle large deformations, optimization is run hierarchically following a coarsetofine strategy powered by image pyramid and control point mesh subdivision. Besides, a postprocessing procedure is proposed to integrate Pareto optimal solutions into a single output, which can improve the estimation accuracy. The observations from experimental results can be summarized threefold. First, our partitioning approach with twoobjective algorithms can obtain deformation parameters more accurately than GA with a single objective. Second, although the fourobjective algorithm performs not as well as expected due to a large number of objectives, it shows to be effective in dealing with complex and subtle deformations. Third, the postprocessing procedure can improve estimation accuracy in many cases. We can observe the usefulness of the proposed method with realworld images.
The main limitation of our method is that high computational resources are required. As future work, we would like to address this issue by further tuning the hyperparameters that can reduce the computational cost without degrading the performance. We are also interested in the referenced point distribution of the NSGAIII. A usersupplied setting may be able to focus solutions on regions that are desirable for the postprocessing procedure.
Declarations
Funding This work is supported by JSPS KAKENHI Grant Number JP20K19568.
Conflicts of interest The authors declare that they have no competing interests.
References
 Multiobjective optimization for deformable image registration: proof of concept. In Medical Imaging 2012: Image Processing, Vol. 8314, pp. 594–600. External Links: Link Cited by: §2.1.
 Recent advances in mesh morphing. Computer Graphics Forum 21 (2), pp. 173–198. External Links: Link Cited by: §1.
 HypE: an algorithm for fast hypervolumebased manyobjective optimization. Evolutionary Computation 19 (1), pp. 45–76. External Links: Link Cited by: §2.2.
 VoxelMorph: a learning framework for deformable medical image registration. IEEE Transactions on Medical Imaging 38 (8), pp. 1788–1800. External Links: Link Cited by: §1.
 Direct estimation of nonrigid registration. In Proceedings of the British Machine Vision Conference, pp. 92.1–92.10. External Links: Link Cited by: §2.1, §2.1.
 Deformable image registration using a cueaware deep regression network. IEEE Transactions on Biomedical Engineering 65 (9), pp. 1900–1911. External Links: Link Cited by: §1.
 Recursively generated Bspline surfaces on arbitrary topological meshes. ComputerAided Design 10 (6), pp. 350–355. External Links: Link Cited by: §5.2.
 PESAII: regionbased selection in evolutionary multiobjective optimization. In Proceedings of the 3rd Annual Conference on Genetic and Evolutionary Computation, pp. 283–290. External Links: Link Cited by: §2.2.
 Nonrigid image registration: theory and practice. The British Journal of Radiology 77 (suppl_2), pp. S140–S153. External Links: Link Cited by: §1.
 Improved NSGAIII with selectionandelimination operator. Swarm and Evolutionary Computation 49, pp. 23–33. External Links: Link Cited by: §2.2.
 An evolutionary manyobjective optimization algorithm using referencepointbased nondominated sorting approach, part I: solving problems with box constraints. IEEE Transactions on Evolutionary Computation 18 (4), pp. 577–601. External Links: Link Cited by: §2.2, §4.
 A fast and elitist multiobjective genetic algorithm: NSGAII. IEEE Transactions on Evolutionary Computation 6 (2), pp. 182–197. External Links: Link Cited by: §2.2, §4, §5.1.

Ranking methods for manyobjective optimization.
In
Mexican International Conference on Artificial Intelligence
, pp. 633–645. External Links: Link Cited by: §2.2.  Direct estimation of nonrigid registrations with imagebased selfocclusion reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence 32 (1), pp. 87–104. External Links: Link Cited by: §1, §2.1, §5.2.
 Image registration by combining thinplate splines with a 3D morphable model. In International Conference on Image Processing, pp. 1069–1072. External Links: Link Cited by: §2.1.
 Realistic cloth augmentation in single view video under occlusions. Computers & Graphics 34 (5), pp. 567–574. External Links: Link Cited by: §2.1, §5.2.
 Adaptation in natural and artificial systems. University of Michigan Press, Ann Arbor, MI. Cited by: §4.
 Direct manipulation of freeform deformations. ACM SIGGRAPH Computer Graphics 26 (2), pp. 177–184. External Links: Link Cited by: §1.
 Evaluation of optimization methods for nonrigid medical image registration using mutual information and Bsplines. IEEE Transactions on Image Processing 16 (12), pp. 2879–2890. External Links: Link Cited by: §2.1.
 Scattered data interpolation with multilevel Bsplines. IEEE Transactions on Visualization and Computer Graphics 3 (3), pp. 228–244. External Links: Link Cited by: §3.
 Manyobjective evolutionary algorithms: a survey. ACM Computing Surveys 48 (1), pp. 1–35. External Links: Link Cited by: §2.2.
 Stochastic ranking algorithm for manyobjective optimization based on multiple indicators. IEEE Transactions on Evolutionary Computation 20 (6), pp. 924–938. External Links: Link Cited by: §2.2.
 An evolutionary manyobjective optimization algorithm based on dominance and decomposition. IEEE Transactions on Evolutionary Computation 19 (5), pp. 694–716. External Links: Link Cited by: §2.2.
 Distinctive image features from scaleinvariant keypoints. International Journal of Computer Vision 60, pp. 91–110. External Links: Link Cited by: §2.1.
 An efficient unified approach to direct visual tracking of rigid and deformable surfaces. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2729–2734. External Links: Link Cited by: §2.1.

ASIFT: a new framework for fully affine invariant image comparison
. SIAM Journal on Imaging Sciences 2 (2), pp. 438–469. External Links: Link Cited by: §2.1.  A probabilistic bitwise genetic algorithm for Bspline based image deformation estimation. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 300–301. Cited by: §2.1.
 Application of evolutionary and swarm optimization in computer vision: a literature survey. IPSJ Transactions on Computer Vision and Applications 12 (3). External Links: Link Cited by: §2.2.
 Templatebased monocular 3D shape recovery using Laplacian meshes. IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (1), pp. 172–187. External Links: Link Cited by: §2.1.
 Medical image registration: a review. Computer Methods in Biomechanics and Biomedical Engineering 17 (2), pp. 73–93. External Links: Link Cited by: §1.
 Fast nonrigid surface detection, registration and realistic augmentation. International Journal of Computer Vision 76, pp. 109–122. External Links: Link Cited by: §2.1.
 Evolutionary multiobjective metaoptimization of deformation and tissue removal parameters improves the performance of deformable image registration of pre and postsurgery images. In Medical Imaging 2019: Image Processing, Vol. 10949, pp. 838–848. External Links: Link Cited by: §2.1.
 Featurebased deformable surface detection with selfocclusion reasoning. International Journal of Computer Vision 97, pp. 54–70. External Links: Link Cited by: §2.1, §2.1.
 Nonrigid registration using freeform deformations: application to breast MR images. IEEE Transactions on Medical Imaging 18 (8), pp. 712–721. External Links: Link Cited by: §1.
 Linear local models for monocular reconstruction of deformable surfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence 33 (5), pp. 931–944. External Links: Link Cited by: §2.1.
 Objective reduction in manyobjective optimization: linear and nonlinear algorithms. IEEE Transactions on Evolutionary Computation 17 (1), pp. 77–99. External Links: Link Cited by: §2.2.
 Face recognition systems under morphing attacks: a survey. IEEE Access 7, pp. 23012–23026. External Links: Link Cited by: §1.
 Nonrigid registration using freeform deformations. Ph.D. Thesis, Technische Universität München. Cited by: §3.
 Freeform deformation of solid geometric models. ACM SIGGRAPH Computer Graphics 20 (4), pp. 151–160. External Links: Link Cited by: §1.
 Deformable template tracking in 1ms. In Proceedings of the British Machine Vision Conference, External Links: Link Cited by: §1, §2.1.

A new ranking method for principal components analysis and its application to face image analysis
. Image and Vision Computing 28 (6), pp. 902–913. External Links: Link Cited by: §6.2.  In defence of RANSAC for outlier rejection in deformable registration. In European Conference on Computer Vision, pp. 274–287. External Links: Link Cited by: §2.1.
 Survey of multi objective evolutionary algorithms. In International Conference on Circuits, Power and Computing Technologies, pp. 1–9. External Links: Link Cited by: §2.2.
 Objective reduction based on nonlinear correlation information entropy. Soft Computing 20 (6), pp. 2393–2407. External Links: Link Cited by: §2.2.
 Deformable surface tracking by graph matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 901–910. External Links: Link Cited by: §1, §2.1.
 Tracking partiallyoccluded deformable objects while enforcing geometric constraints. arXiv preprint arXiv:2011.00627. Cited by: §1.
 Multiple nonrigid surface detection and registration. In IEEE International Conference on Computer Vision, pp. 1992–1999. External Links: Link Cited by: §2.1.
 MOEA/HD: a multiobjective evolutionary algorithm based on hierarchical decomposition. IEEE Transactions on Cybernetics 49 (2), pp. 517–526. External Links: Link Cited by: §2.2.
 A new dominance relationbased evolutionary algorithm for manyobjective optimization. IEEE Transactions on Evolutionary Computation 20 (1), pp. 16–37. External Links: Link Cited by: §2.2.
 Fast affine template matching over galois field.. In BMVC, pp. 121.1–121.11. Cited by: §2.1.
 Robust projective template matching. IEICE TRANSACTIONS on Information and Systems 99 (9), pp. 2341–2350. External Links: Link Cited by: §2.1.
 MOEA/D: a multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation 11 (6), pp. 712–731. External Links: Link Cited by: §2.2.
 Enhancing MOEA/D with information feedback models for largescale manyobjective optimization. Information Sciences 522, pp. 1–16. External Links: Link Cited by: §2.2.
 A fast 2D shape recovery approach by fusing features and appearance. IEEE Transactions on Pattern Analysis and Machine Intelligence 31 (7), pp. 1210–1224. External Links: Link Cited by: §2.1.
 SPEA2: improving the strength Pareto evolutionary algorithm for multiobjective optimization. In Evolutionary Methods for Design, Optimization and Control with Applications to Industrial Problems, pp. 95–100. Cited by: §2.2.
Comments
There are no comments yet.