Truncated singular value decomposition in ripped photo recovery

Singular value decomposition (SVD) is one of the most useful matrix decompositions in linear algebra. Here, a novel application of SVD in recovering ripped photos was exploited. Recovery was done by applying truncated SVD iteratively. Performance was evaluated using the Frobenius norm. Results from a few experimental photos were decent.


Introduction
Singular value decomposition is a matrix decomposition that decomposes or factorizes a matrix as a product of three component matrices. It was established ways back in the 19th century due to the contributions by Eugenio Beltrami, Camille Jordan, James Joseph Sylvester, Erhard Schmidt, and Hermann Weyl [1].
Singular value decomposition always appears in problems that entail analysis of large amount of data because data often comes in the form of matrix. The application of SVD has been well known in principal component analysis, data and image compression, and in solving least square problems via pseudoinverse [2][3][4]. Its application is very board applicable to a large scope of problems especially when the problems are representable in matrix and involve matrix approximation. Other SVD applications have been found in areas such as dynamical systems, genomics, textual searching, robotics, financial mathematics and in many others. Interested readers may refer to, for example, [5] and references therein. However, application of truncated SVD in ripped photos recovery is not widely reported in the literature.
In the literatures, ripped recovery was commonly treated as missing data interpolation. The pixel values on the ripped regions were considered lost, so recovery was done by filling in the values using the embracing data. Various professional filling strategies had been reported, for examples, in [6][7][8][9]. The objective of this study is to highlight the missing data treatment under the SVD paradigm which is simpler though not as much robust.
in the interval [0,1]. This fact makes matrix operations, such as SVD, play a role in photo processing. Prior to the exposition of the iterative truncated SVD ripped recovery approach, a very brief description on SVD is given.

The SVD
For a given matrix of size × , the singular value decomposition of is a factorization where is × , is × , and is × . Both and are orthogonal matrices while is diagonal and its diagonal entries are known as the singular values of . The singular values are positive and arranged in descending order such that = diag( 1 , 2 , … , ) and The square of are the eigenvalues of (and ). For a matrix of rank , will have leading nonzero diagonal entries. The columns of and are called the left and right singular vectors of and are the eigenvectors of and respectively. The column of , the column of , and the singular values of stisfy Sometimes, it is useful to regard equation (1), the SVD of , as an outer product expansion (somewhat analogous to series expansion) of in terms of the singular values and the columns of and : Here is the rank of the matrix and is defined as the dimension of the column space of . In this context, can be regarded as the number of nonzero singular values of .

The truncated SVD and the low-rank matrix approximation
The -truncated SVD of a matrix is done by keeping only the first ≤ nonzero singular values (and hence the first columns of and ) such that The resultant matrix , known as the rank-approximation to , is just an approximation of the matrix since ( − ) nonzero singular values are discarded.
Since SVD yields descending singular values, the truncated SVD, which only packs the first few largest singular values, will give an approximation that encapsulate the most essential weightage.

The iterative truncated SVD ripped recovery algorithm
The main function in ripped recovery is to dissipate the tear lines meanwhile maintain the natural look of the photos. As a pre-process, the tear lines on the digitized ripped photo were manually erased. These erased regions were recognized as the flaw region and their location were stored. Thus the image matrix consists mostly the perfect region and a minor portion of flaw region such that = (perfect) ∪ (flaw). The iterative truncated SVD recovery idea is described in the following algorithm.
Step 1: The image matrix undergoes a -truncated SVD to obtain its low-rank approximation . As in , there are perfect and flaw regions in also such that = (perfect) ∪ (flaw).
Step 2: The matrix entries of within the flaw regions are all updated by their corresponding approximated entries of while the rest of the entries of , in the perfect regions, are kept unchanged. This produces ̃= (perfect) ∪ (flaw).
Step 3: The Frobenius norm of −̃ is computed to check against the preset tolerant . If | −̃| > , then is replaced by ̃ and go back to Step 1; else proceed to the next step.
In each iteration, the low rank approximation of the previous image matrix is computed. Due to low rank approximation, the pixel value in the flaw region would spontaneously blend with the surrounding pixel values coming from the flawless region. This kind of iterative blending created certain interpolation effect that made the tear lines faint away. And during the process, the truncated SVD that was capturing the most significant characteristics of the photo would infer the pixel value to restore the photo.

The performance measure and the Frobenius norm
As an objective performance measure, the grey level differences between the undamaged photos (assume available here) and their recovered versions were computed by adopting the Frobenius norm. The Frobenius norm of a × matrix is defined by It is in fact the square root of the sum of the absolute squares of all its entries.
To estimate how good the recovery is, the Frobenius-based similarity level between the undamaged and the recovered image matrices is defined where 0 is the image matrix of the undamaged photo, is the image matrix of the ripped photo, ̃( ) is the image matrix of the recovery after -th truncated SVD iterations. The fraction in the equation ( can be thought off as the relative error. For a perfect recovery, ̃( ) = 0 , there will be no difference between the recovery photo and the undamaged photo. Hence generally, the closer the value of to unity, the better the recovery will be.

Experiments and discussion
The proposed iterative truncated SVD approach was experimented on three test photos. All the test photos in this study were "synthetic" such that their undamaged versions (before ripped) were available. The three test photos were arbitrarily chosen to be (i) portrait, (ii) scenery and (iii) fruits. Only grey scale photos were considered here.
To show the recovery capability of the iterative truncated SVD approach, the undamaged, the ripped, and the recovered versions of the three test photos, lined up from left to right in series, were shown in Fig. 1, Fig. 2, and Fig. 3, one after another.   In every iteration during the truncated SVD process, the recovery progressed gradually. As a sample, the changes in the third test photo (fruits) during the first few iterations were displayed in Fig. 4.   Fig. 4. The changes in the third test photo (fruits) as the truncated SVD iteration progressed.
Though the iterative truncated SVD managed to make the tear lines to be less obvious, the recoveries were not seamless. Artifact was still noticeable especially under a large zoom in.

The similarity level
The Frobenius-based similarity level as defined in equation (6) was computed for each test photo and was listed in the following table. In fact, the Frobenius-based similarity level is not a practical assessment because the undamaged photo must be available for its calculation. So, there will be no qualitative measure in real application, it all depends on naked eye observation to determine how natural looking the recovery is.

Photo
Similarity level Portrait 0.627922 Scenery 0.704459 Fruits 0.643498

Conclusion
Apart from aesthetic appeal, ripped photo recovery may be meaningful when the photo has great historical value. Besides, recovering ripped photos may play a major role helping crime inspection in forensics. An interesting application of SVD in ripped photo recovery experimented on three samples had been presented here. The algorithm of the iterative truncated SVD was rather simple yet it produced encouraging visual restoring quality which was slightly to see. Again, this seems another manifestation that SVD is ubiquitous in the area of image processing.