Virtual Dressing using Augmented Reality

With the rise in Virtual Reality technology now it is possible to bring the same experience of offline shopping in the online platform. Virtual Dressing using Augmented Reality is a platform where users can try clothes virtually. As users cannot physically try the dress in online mode this application will help the user to make a better judgement on how the dress will look on him. The proposed system will consist of an application where users can create their 3D model with just their pictures and can apply the garment on their model, which can be viewed in AR space to make a judgement of how the model looks on a person.


Introduction
Due to the pandemic situation, there is a rise in online shopping platforms. People engage more in online shopping. But in the clothing section, users hesitate to buy as they will not get the same experience as with offline shopping because they can't try the clothes in online mode. This is a problem for both, the customers and the sellers. So, there is a need for a virtual trial room for online platforms.
With the rise in Virtual Reality and Augmented Reality technology now it is possible to bring this functionality to the online platform. A system could be devised where users can create their 3D model with just their pictures and can apply the garment on their model, which can be viewed in AR space to make a judgement of how the model looks on a person. Furthermore, we can suggest the appropriate measurement which fits the person. 3D virtual garment model design has gained large popularity in the computer-aided design (CAD) community. It is very important in the development of high-quality computer-generated video games since the quality of garment models on characters can play an important role in the quality of the game. With recent advancements in clothing simulation, it is possible to give dynamic behaviors to virtual garments. The garment model is designed to fit a virtual human body as a separate mesh on top of it.
2 Related Works M.R. Minar et al. [1] propose an approach to try clothes on a 2d image of a person. Here a common processing pipeline for this setting has two stages: first, the try-on cloth is warped around the target human (GMM), and then the warped cloth in blend according to the pose of human (TMM). The cloth warping stage obtains the complete body silhouette area from the target human image. Then other body parts are masked leaving just the target clothing area. Further human parts are retained in the blending stage, the warped cloth mask is applied to the target cloth area. The first iteration of this model was proposed by Bochao Wang et al. [2] as CP-VTRON further was refined and called CP-VTRON+.
N. Wang et al. [3] proposes a method to transfer clothing texture on human avatars called Pix2Surf. Pix2Surf is capable of transferring the texture of clothing images to 3D human models in form of 3D garments. These 3D garments are worn on top of the 3D human model. This model helps to transfer the texture from the front and back garment images to the garment mesh. Pix2Surf maps the garment texture selected from a large number of clothing items available on online websites onto the human model in real-time. This model is useful to generate a virtual try-on.
S. Saito et al. [4] propose an effective representation that aligns the pixels of 2D images corresponding to the 3D object. This method is called PIFu(Pixel-Aligned Implicit Function). It maps the dress onto the 3D object. PIFU is an end-to-end deep learning method used to form a digital clothed avatar. PIFu infers the underlying 3D surface and texture from a single image or can opt for multiple images. Highly detailed shapes, such as clothing and variations can be digitalized uniformly. T. Alldieck et al. [5] propose an effective model to reconstruct people's clothing from a single RGB image. Using it, PIFu is capable of generating high-resolution surfaces such as the back of the person from the single front view. This method achieves a state-of-the-art performance and outperforms the prior work of clothed  The main objective is to design a service for e-commerce fashion websites. The Figure 1 shows the overall System architecture. The fashion websites will provide the customers with the try-on option on their website. The fashion website will send user images along with the garment image to the back-end which then processes the user and garment image to append garment on the user image and output the 2D image and 3D user model to the fashion website.

Methodology
.In the prescribed method, the garment image is applied on a user image then pipeline the user image to generate a 3D model. So, it consists of two separate parts: Garment on 2D image and transforming 2d image into 3d model. The first part i.e garment on 2D image is based on the CP-VTRON+ model. Before applying CP-VTRON+ [1] the pre-processing should be done using CIHP_PGN [7] to segment the input user image. Further, the user pose is determined using Openpose [8] so the heatmap can be drawn to identify different body parts of the user image. The CP-VTRON+ has two stages: The clothing warping stage and the Blending stage.
The cloth warping stage is used to separate the target clothing area from other body parts and generate the pose structure to blend the cloth according to the user pose. The blending stage retains the original body parts and fits the wrap cloth onto the target clothing area. The model is based on CNN geometric matching [9]. The second part is based on the PIFu model. In PIFu(Pixel-Aligned Implicit Function) the task is to generate the complete 3d geometry structure using a single user image. Here the output of the CP-VTRON+ model is fed to PIFu as an input. Given an input image, it reconstructs the surface geometry and predicts the textured surface of a clothed human. Further, the texture is preserved by storing the RGB values of the image and transforming them into RGB vectors which are applied using the ray-tracing algorithm.

Scope
Virtual dressing using Augmented Reality is a platform that can be integrated into different e-Commerce and fashion websites. The basic motive is to give the people a tool using which they will be able to see the real-time simulation of different clothes without having to wear them physically. It's a Web-based application integrated into another website or can work as a standalone application to create a human 3D model. This system automatically transforms 2D pictures into a 3D model. This model is further used by users to try clothes on different websites. This model can be viewed in AR space to take a better judgment on the looks of the clothes.

Dataset
The images used for training the model are from VITON [10] and renderpeople website [11]. The Figure 2 shows dataset consists of cloth images. Figure 3 shows the target human images.

Data-Preprocessing
The cloth images are preprocessed by CIHP-PGN [7] network to get a cloth mask. The background of target human images is removed using removebg API [12]. The target human image along with the human image mask shown in Figure 4 is passed onto Openpose [8] to generate a pose heatmap shown in Figure 5.  The model creates a pipeline of the process of creating a 3D model from a 2D image. The Figure 6 shows the flow of execution. The dataset provided by VITON [10] is processed by CIHP-PGN to generate the image mask and Openpose to generate a human pose map. Removebg is used to remove background from cloth and target human image. The cloth image along with cloth mask and a human image along with pose image is passed to CP-VTRON+. The pose grid is used to generate a warp cloth image in the cloth warping stage. The wrap cloth is appended on the target human image in the blending stage. Further, the output image from CP-VTRON+ is scaled to 512x512 pixel resolution to pass into the PIFu model. The PIFU model is used to generate the 3d model of the human image.

Login and Register Page
The user registers using valid email id and password shown in Figure 7. The login interface where the user puts valid email id and password to login into application shown in Figure 8.

Home Page
The main dashboard of the application. Here the user can upload the image as well as view the original uploaded image. The default image is provided so user can try-on a default image shown in Figure 9. The showcase gallery of garment where the user can select the garment to be worn on the image shown in Figure 10.

Result Page
The output of the model shown in Figure 11 is viewed as a 2d image as well as a 3d model. The 3d model is rendered using Threejs [13] library so the user can freely rotate the image in the 3d view.

Conclusion
The primary motive of the paper is to solve the user problems while shopping on the online fashion platform. As the user cannot try clothes in online mode so they do not get the same experience as offline mode. Hence this will bridge the gap by giving the user an option to try the cloth on their image and the user can get the feeling of wearing them without actually trying the cloth. Further, the user can take a look in 3D space at how the cloth looks on the image. For now, this product is only addressing the issue of the shirt but further, it can be extended to support all kinds of clothing accessories.