Aerial assessment of solar panel surfaces using drone

. The concept of image processing has come a long way in the recent past. There are multiple softwares based on image processing with better accuracy and efficiency even when compared to the human eye. With open-source libraries like Python growing rapidly in development, there are many image processing libraries like OpenCV, Mahotas, Pillow / PIL, NumPy are being used widely and in most image processing softwares. This project is a demonstration of how image processing and machine learning is used to identify and differentiate different contaminants on the surface of a solar panel. A solar panel is surrounded by lots of empty areas like on rooftops of buildings and solar farms. Due to this, it is very easy for contaminants like dirt and bird droppings to make the solar panel dirty. When this happens and with more accumulation of contaminants, the solar panel efficiency exponentially decreases. Hence it is important to keep such surfaces clean as much as possible. Although, cleaning the surfaces is a tedious and moreover, a dangerous job. It is better to be prepared with the nature of the contaminants on the surface before actually going to clean it so that the cleaners will not have to take anything more than the necessary equipment.


INTRODUCTION
Drone technology is evolving at an alarming rate. They are used in almost every single field today from security and surveillance to even movie filming and content creation. Purely because of its agility and stability, drones have made place for themselves in today's society.
Paired with the power of image processing, drone technologyis used in research, scouting inaccessible areas, reachingincredibly high places and many more applications. Drones are also used to scout old buildings and identify cracks, weaknesses, areas where paint has not been put properly usingsimilar technology.
Using the above applications, this paper talks about how drones and an integrated camera can be used to survey solar panels and scan for the constituent contaminants, which if left unattended will reduce the efficiency of the solar panel exponentially. [1] This paper talks about the importance of image segmentation and its varied application in today's world ranging from medical image analytics, surveillance of video, compression of images, augmented reality etc. The success of Deep learning (DL) has helped the growth of image segmentation techniques which go hand in hand with various DL models. It further covers developments of instance and semantic segmentation along with encoderdecoder architecture, GAN models, convolutional pixel label networks and many others. [2] This paper talks about a novel design of a Smart Solar Panel (SSP) system which improves upon the existing PV solar panel system. To produce optimal energy efficiency, the solar panel has to be kept facing towards the sun and the panel should be free of any contaminants. This is achieved by robotic functionalities which would move the panel according to the current position of the sun and would remove contaminants present on the panel. This combined functionality would thereby provide a SSP system which would be simple, clean and highly effective. [3] This paper addresses the issue of lack of appropriate training data or an uneven class balance within datasets. Data augmentation is given as one of the solutions in addressing this issue where image transformation is done along with style transfer and generative adversarial networks. It then talks about a new method of data augmentation based on image style transfer. The method allows generation of new images of high visual quality combining the content of a base image with the appearance of other ones. The fresh crop of images thereby pre-trains the given neural network increasing the efficiency of the training process. [4] This paper talks about incorporating a novel data augmentation method using the concept of image transformation. New data points are generated by adding noise to lessen sparsity in a particular feature space. Then images are correlated wrt to the new data points. The uniqueness in this method lies in creating an image processing tree where small number of instances are used for generating new data. By doing so, the classifiers become more robust and sparsity is reduced. [5] This paper examines the advantages of incorporating an augmented dataset with artificially produced images aka synthetic images data into the training of a machine learning classifier. Data warping, which generates additional samples through transformations done in the data-space, and synthetic over-sampling, which generates additional samples in the feature-space, are two methods for generating more training samples. [6] This paper talks about a Panel cleaning system which can be implemented due to the rising level of dust and other contaminants in a solar panel system. The techniques implemented are AI and CV (Computer Vision). This is carried out by a overhead camera which monitors the panels and an cleaning apparatus to clean the panel. The implementation of ML techniques would give more insights on the types of contaminants present and the preventive measures which can be implemented in the long run. [7] This paper talks about the rise of solar power over conventional energy sources like coal, gas and oil. This therefore requires vast solar based energy facilities. These facilities usually occupy open areas which leads to accumulation of dust and other contaminants in it, leading to loss of efficiency. This paper therefore analyzes the efficiency of solar panels under different contaminants and how it may influence its working.

METHODOLOGY
This project is an amalgamation of hardware and software components. The hardware component revolves around the drone itself, its construction and working. The software component revolves around the algorithms which each image is passed into. The software component is split up into three sections; image processing, machine learning and image augmentation.

Image Processing
Every picture which the drone captures is passed through an image processing algorithm. This algorithm is written using Python's OpenCV and numpy. OpenCV is an image processing library written in Python. We first pass the image into a function which converts the image into its HSV equivalent image. Then, we create a mask using the lower and upper HSV for brown color. Now, we have all the possible colors which dirt can take up on the solar panel. We then pass the HSV image into a bitwise AND function so that all the colors which do not fall within the brown color's range are given a value of 0 (which is a black pixel). After this is done for every single pixel in the image, We get an image with only the brown colored dirt spots from the solar panel surrounded with black pixels. The dirt has been successfully masked out.
The similar process is done to mask out the solar panel's surface as well. The only change is instead of using the lowerand upper bounds of brown color, we change it to the lower and upper bounds of blue color; as all the surfaces of solar panels are generally blue.
Once we pass the image into the new mask, the result is a similar image, with only the blue parts from the solar panel surrounded with black pixels.
Both the masked images are passed into a bitwise OR gate to get a final image with only the surface of the solar panel and its residing contaminants. Then, the number of brown pixels is counted and divided by the number of total colored pixels to get the total percentage of contaminants on the solar panel surface.

Machine Learning Algorithm
One of the largest investments in renewable energy that significantly improves environmental protection is solar energy. Regarding this, the industry for solar energy is fully aware of the necessity of maintenance issues that foresee the energy loss of solar panels and take precautions. The energy loss of PV (photovoltaic) panels can be influenced by a number of factors, including shadowing, dust, temperature, and a direct reflection of the sun. The buildup of snow, sand, grit, leaves, pollen, and bird droppings on solar panels is referred to as "soiling," and this can easily result in a daily power loss of more than 1% [2].
Our research intends to distinguish the kind of contaminant on the PV panels for solar energy systems based on natural and environmental factors. A convolutional neural network (CNN), a powerful method for image classification in computer vision, is set out with a pretrained model to classify panel images in the training. The dataset splits into categories that comprise 2 types of contaminants, bird dropping and dust, as shown in Table 1.

Dataset
In our initial research we found that there is a dataset which is called the Deep Solar Eye which has 45000 images of soiling the panel but this could not satisfy our requirements. It not only slowed down the training but also computationally inefficient. We then decided to create a customized dataset consisting of only required classification i.e bird dropping and dust on solar panels.
An augmented image dataset of images was therefore created with various image transformation operations applied on it. Two original images were used:-One image was dust on a solar panel and the other image was bird dropping on a solar panel. A 100 images were created from each image (200 in total) therefore creating an image dataset which was further used for training the model and hence classification.

Process
Extraction of features from an image is required for image classification in order to spot some trends in the dataset. Convolutional neural networks (CNN or ConvNet) are a subclass of neural networks that are mostly employed in voice and image recognition applications. With no loss of information, its integrated convolutional layer lowers the high dimensionality of images. CNNs are therefore very well suited for this use case.
VGG was employed in this categorization as the CNN architecture. It was developed by the Visual Geometry Group at Oxford, hence the name VGG. It incorporates and builds on some of the ideas from its forerunners and uses deep convolutional neural layers to increase accuracy. A variation of the VGG model called VGG19 has 19 layers in total (16 convolution layers, 3 Fully connected layer, 5 MaxPool layers and 1 SoftMax layer).
• This network received a fixed-size (224 * 224) RGB image as input, indicating that the matrix was shaped (224,224,3). • The only pre-processing that was carried out was the subtraction of each pixel's mean RGB value from the entire training set. • We covered the entire idea of the image by using kernels that were (3 * 3) in size with a stride size of 1 pixel. • To maintain the image's spatial resolution, spatial padding was applied.
• Max pooling was carried out with stride 2 over a 2 by 2 pixel window.
• Tanh or sigmoid functions were utilised in earlier models, thus Rectified linear unit (ReLu) was used to introduce non-linearity to improve the model's classification and computation time. This proved to be much superior than those.

Outcome
Tests on 197 photos that had all the classifications were conducted to assess the proposed model. After ten epochs, the training process came to an end. The methodology's classification report is displayed in Table 2. It has been demonstrated that the 95% accuracy rate for picture classification is very good.

Table 2. Classification report
The classification performance evaluation is shown in Fig. 2, which expresses comprehensive results. We have the following observations regarding the prediction outcomes in the classification report and the confusion matrix. First off, the dataset's images share a lot of properties, such as the colour of the panel, where the shots were taken, and how the soiling appears, which makes the suggested model highly accurate. Second, there isn't much soiling in the image under the "dirty" category, and a number of the images are somewhat similar. The results enable the prediction of their category membership. Finally, the image belongs to the right category if it includes a sizable area while several various types of soiling are occurring simultaneously on the board.

Image Augmentation
It is a technique, which is used to artificially expand the size of a training dataset by creating different versions of the image in the dataset. Image transformation in these scenarios use various operations wherein the images are flipped, rotated, sheared, mirrored, zoom and many more such operations. The augmentation techniques vary according to the existing dataset and the domain in which the problem exists.
Deep learning Techniques particularly CNN (Convolution Neural Network model) therefore creates a network of images which are interlinked with each other and the CNN also learns in identifying the image irrespective of any transformations on it also known as "Invariance in the Imagelocation".
One of the popular techniques is "Image augmentation using Keras", Where the training examples are augmented using random transformations so that each image is different and it also prevents overfitting (where the image fits exactly with the training data). "Keras. preprocessing.image.ImageDataGenerator" class helps to achieve 1. Random image transformations and its associated operations.
2. Instantiates the Generator of augmented image batches.
clockwise and anticlockwise in order to give a total downward thrust.
(iv) Power supply: the drone is powered by a 4500mAh battery which powers the rotors and pixhawk. Flysky FS-iA6b receiver: the receiver is used to receive signals from the flysky remote (v) Flysky remote:The Flysky remote is a battery operated remote which gives the drone the signals for flight, during flight and for landing.
{vi} Mission Planner Software: This software is made specifically for the Pixhawk Flight controller. It shows information pertaining to the location, calibration, armament and readiness of the drone.
Below is a depiction of how the hardware component of the drone are all connected The below figure is a picture of the completed drone which has achieved successful flight Using the concept of image processing, the algorithm has successfully been able to determine the percentage of contaminants which resides on the surface of the solar panel. The accuracy of this result is achieved as the algorithm first converts the image into its HSV equivalent rather than processing on its RGB version, which makes it easier to differentiate between colors. The drone has been built using the parts listed above and has achieved successful flight. Machine learning achieves good performance with an average accuracy of 0.98%. This paper's main contribution is to support the determination of the countermeasure by knowing the soiling feature. Moreover, this implemented model gives a proper solution to help to recognize soiling on solar panels automatically, thereby helping periodic maintenance increase overall efficiency. The image augmentation technique therefore creates a dataset of images from a given image which can be therefore used to expand the size of the training set.
The number of images to be generated and the different operations to be performed on the images is user specific making it very effective in creating datasets.

CONCLUSION
This project has been constructed in a manner so that the process of cleaning solar panels becomes a lot more efficient and safer. Using the technology described in this paper, cleaners and engineers can assess the surface in a much more precise and elegant manner. Cleaners can first scout the surface of a panel and be a lot more prepared with their equipment and their cleaning methodologies using this technology. Using the concepts of image processing and machine learning, the results are very accurate as well. The models have been trained using multiple custom datasets which were developed using image augmentation.