Color Image Inpainting By an Improved Criminisi Algorithm

Due to the incorrect filling order and the fixed size of patch, the traditional examplar-based image inpainting algorithm tends to cause the image structure fracture, texture error extension and so on. So in this paper, it proposes an improved Criminisi algorithm with adaptive adjustment with gradient variation to color image inpainting algorithm. Firstly, to overcome the discontinuity of the edge structure caused by the incorrect filling order, using curvature of isophotes to constraint the filling order. Secondly, in order to solve the lack of the step effect in rich texture region, it adaptively adjusts the sample patch size according to the variation of local gradient. Finally, the local search method is used to find the best matching patch. The experimental results show that the proposed algorithm’s PSNR increased by 1-3dB and obtain better results in terms of different types of images.


Introduction
Digital image restoration is a process of filling unknown area according to the information of known area of the image in a visually plausible way. One of the most representative algorithms is the exemplar-based image inpainting algorithm proposed by Criminisi et al. [1]. In recent years, many scholars have put forward different improvement methods aiming at the deficiencies of the Criminisi algorithm [2][3][4][5][6][7][8]. Xiang et al. [4] inducted the curvature to the data item so that the information could be extended along the direction of the isophotes, and introduced the matching method based on matrix similarity into image restoration to solve the image visual discontinuity problem caused by error matching. Qi et al. [5] changed the priority function to the weighted sum of the terms, avoiding the order of error filling when the data item is zero; Considering the existence of some mistakes between the patch to be repaired and the matching patch, the algorithm improved the update of confidence . The improved algorithm had a better repair for the image structure, but in the rich texture region produced block effect. Yin et al. [6] added the curvature of the isophotes to the calculation of the priority function, and led the curvature term into the search matching, which had a better restoration effect on the image with single structure, but it was not ideal for image with complex structure and texture. Deng et al. [7] completed the image restoration in phases by defining a new priority function, and searched for the matching patch using local search to find the best matching patch to reduce the repair time. Wang et al. [8] used the adjustment factor to determine the priority of each patch and added a normalized cross-correlation function (NCC) to the matching condition to scale the similarity between the source patch and the target patch.
However, these algorithms rarely take into account the impact of the priority calculation when the data item becomes zero in the priority function and the influence of curvature on the filling order, which is easy to produce the wrong filling order during the restoration process. Moreover, most of them use a fixed sample patch size to search for the best matching patch, thus in the rich texture region it will produce the staircase effect. So in this paper, the curvature of the isophotes is introduced as a cofactor into the calculation of the priority function to make the filling order more accurate and to prevent the error extension or discontinuity of the image structure. In addition, the algorithm adaptively adjusts the sample patch size according to the local gradient variation of the image. Finally, the local search is used to replace the global search in traditional inpainting algorithm. The improved algorithm makes the entire repair order more reasonable, which has better repair effect of the image structure, especially on the curve part.
The paper is organized as follows. Section 2 introduces the Criminisi algorithm, analyzes the shortcomings of the traditional algorithm and proposes an improved algorithm. Section 3 gives the comparison of experimental results between the improved algorithm and other algorithms. Finally, the conclusion is described in section 4.

Criminisi Algorithm
The Criminisi algorithm [1] is an exemplar-based image inpainting algorithm, which combines the characteristics of transmission structure and texture synthesis. As shown in Fig. 1, assuming that the target region is indicated by Ω and . The Criminisi algorithm contains the following three steps [1] : First, the priority function ) ( p P of target patch p ψ is the product of two terms: the confidence item ) ( p C and the data item ) ( p D .They are defined as follows respectively: Second, when all priorities of target patch on the fill boundary have been completed, it looks for a patch p ψ with the maximum priority, and then uses the global search to find the most similar sample patch q ψ in the source area. represents the sum of squared differences (SSD) of the known pixels in the corresponding patches.
Repeat the above steps until the pixels of the area to be repaired are filled completely.

The Priority Improvement
In the practical application, it is found that when the data item of some pixels may be zero in the filling process, its priority value are also zero, which leads to incorrect filling order and affects the repair effect to a certain degree. In addition, the curvature of the isophotes [9] as a second order differential is one of the most important descriptions of the morphological features of the isophotes reflecting the local characteristics of the image [10]. However in the algorithm [7], they did not take into account the above problems when calculating the priority. Therefore, this paper adds the curvature of the isophotes as a cofactor to the calculation of the priority function. On the one hand, the structural information of the image can be extended along the direction of the isophotes so that the structural part can be repaired better, which can prevent the error extension or discontinuity of the image structure. On the other hand, when the data item of some pixels are zero, it can avoid the wrong filling order. The improved priority function is defined as follows: , it achieves the inpainting of the texture part of the image. At this time, the initial value of ) ( p C is still set according to (4).

Adaptive Adjustment of Sample Patch Size
For the traditional exemplar-based image restoration algorithm, a fixed-size sample patch is used to search for the best matching patch. However, an image usually contains different edge structures and texture features, if using the same size of sample patch for image restoration, it will greatly affect the overall image restoration effect. In order to overcome this shortcoming, this paper adaptively adjusts the sample patch size by using the gradient change of the local area of the image. If the image has complex background, in order to prevent the wrong extension of the edge structure, using a smaller sample patch to search for the best matching patch. When the image of the texture area is relatively flat, then using a larger sample patch to avoid the staircase effect. The specific implementation method is as follows: Firstly, assuming that the number of pixels of the color damaged image is M , and convert the color image to the gray-scale image I according to (  9× .

The New Searching Strategy
According to the neighborhood of the image with the similarity principle, the proposed algorithm uses local search method to determine the best matching patch in order to improve the inpainting efficiency. The main idea is as follows: at the center of the pixel p with the maximum priority, expanding Z pixels through up and down, left and right. That is, it forms a relatively large patch, which size is ( ) ( ) Where num is the total number of pixels of damaged area.
As described above, the steps of the improved algorithm are as follows: 1) Select the target area Ω , and marks the area with a specific color.
2) Determine the fill front Ω δ of the target area Ω , when φ δ = Ω , the algorithm ends and outputs the restored image.
3) Convert the color image to the gray-scale image I according to (10). ψ is searched in the size of ( ) ( ) of the local known area. 7) Copy the image information of the best matching patch q ψ to the target patch p ψ . 8) Update the confidence value according to (7). 9) Repeat 2) to 8) until all the target region to be filled.

Experimental Results and Analysis
In order to verify the effectiveness of the proposed algorithm, the simulation experiment is realized with Matlab R2011 on the computer with the 3.30 GHz processor and the 2GB memory. Fig. 2 shows the results of using the Criminisi algorithm, the algorithm [6], the algorithm [7] and the improved algorithm to restore the Barb image. As show in Fig. 2(c), 2(e) and 2(f), these three algorithms have better repaired results for pure texture regions. In algorithm [6] and this paper, the curvature of the isophotes is added to calculate the priority. So there is no structural discontinuity at the junction of the arms and knees in Fig. 2(d) and Fig. 2(f). While the algorithm [7] and the Criminisi algorithm doesn't introduce the curvature into the priority. Therefore by comparison of the inpainting results can be obtained that introducing the curvature of the isophotes will be more accurate to calculate the priority function. Moreover, the inpainting result of the proposed method provides better visual quality and the PSNR value increases by about 3dB compared to other algorithms.   . Fig. 3 shows the inpainting results of the Peppers image. As show in Fig. 3(c), the Criminisi algorithm has some small flaws in the inpainting process. In Fig. 3(d), the algorithm [6] is not good for the restoration of the edge structure of the image, which appears obvious structural discontinuity.
While the results of the image inpainting between the algorithm [7] and the improved algorithm is much better than the algorithm [6]. Fig. 4 presents the comparison of the restoration results for text removal. From the results of the restoration, it can see that when the image with complex background structure is repaired, the inpainting effect of this algorithm is visually better and without the wrong texture filling. Table 1 shows the comparison of the PSNR value of the proposed algorithm and other algorithms. From Table 1, the proposed algorithm's PSNR value has increased significantly than other algorithms and has a better visual effect.

Conclusion
Owing to the incorrect filling order, the traditional examplar-based image inpainting algorithm leads to the image structure fracture and texture error extension. In this paper, an improved color image restoration algorithm is proposed based on the Criminisi algorithm. Firstly, the curvature of the isophotes can be used to characterize the local information of the image. Therefore, the curvature is used as a new control factor to redefine the priority of the block, so as to ensure that the filling order is more accurate and avoid structural information error extension and enhance the continuity of the image structure. Secondly, by using the local gradient and combining with the mean and variance of the gradient of the image, the size of the sample patch is adaptively adjusted to overcome the staircase effect caused by the fixed size sample patch. Finally, in order to improve the overall efficiency of the image restoration, this paper uses local search method to find the best matching patch. The experimental results show that the proposed algorithm can effectively overcome the shortcomings in the traditional algorithm, and achieve better restoration effect for the damaged area of different degrees of image, especially for the damaged image with obvious structural changes. Since the proposed algorithm uses adaptive adjustment of the sample patch size, the running time has increased compared to the Criminisi algorithm.