Novel GAN-Based Image Completion: Addressing Structure and Texture Consistency in Missing Regions
Abstract
The use of Deep Neural Networks (DNNs) to solve Image Completion (IC) has emerged as a popular research topic, as this study demonstrates. Completion algorithms must handle structure and texture properly in order to generate realistic results because they are two essential components of images. To fix an image, several modern techniques employ the end-to-end framework, which ignores texture and structure in particular. From the outcomes, deformed structures and uneven textures are frequently obtained. The sketch completion network and a texture completion network are contained in a novel IC method is suggested. The objective of Generative Adversarial Network (GAN) is to restore the sketch structures in the missed portion of an image. By representing the two components separately in a DNN, the proposed approach not only successfully synthesizes semantically valid and visually reliable data in the missing region but also allows a user to change the structure characteristics in that region dynamically. Graph Neural Network (GNN) creates consistent texture data in the missing area with the sketch output and the surrounding partial image.
Keywords:Image completionDeep learningGenerative adversarial networksTexture synthesisStructure
reconstruction, Convolutional neural networks.
1. Introduction
Several applications like image editing, restoration and manipulation have made Image Completion (IC) vital. IC is also named as inpainting, and it is a widely researched problem in the Computer Vision (CV) [1]. By focusing on filling in missing or corrupted parts of an image seamlessly, image inpainting leverages powerful algorithms to recreate or reimagine visual content in a way that appears natural and undisturbed.
Image inpainting has many practical applications, such as restoring old photos, removing unwanted objects, and filling missing areas in historical artifacts [2]. Then, areas from the known sections of the image are copied to the missing areas in traditional IC methods such as diffusion-based methods and patch-based techniques. In complex situations, these techniques frequently fail, particularly where the missing parts require semantic understanding or involve complicated textures and structures. A type of Deep Learning (DL) method named Convolutional Neural Networks (CNNs) has revolutionized the field of IC [3]. By using big datasets to train rich feature representations, deep networks have demonstrated the ability to provide outputs that are more context-aware and semantically important.
For the purpose of managing pixel data, CNNs are very effective for Image Recognition (IR) and Image Processing (IP) tasks. With the help of enormous volumes of training data, it allows the network to identify intricate patterns and features [4]. For applications that require a deeper understanding of the image global context specifically (Fig. 1), thus the suggested method performs well when compared to other methods.

For Image Generation (IG) tasks and IC tasks, a type of DL named Generative Adversarial Network (GAN) is effective [5]. A GAN consists of the game-theoretic simultaneous training of two competing networks, the Discriminator (D) and the Generator (G). In a GAN, the D is just a classifier. It attempts to differentiate between data generated by the G and real data. This adversarial training process enables GANs to generate highly realistic and contextually accurate images, even when significant portions of the image are missing [6]. GANs can represent intricate textures, structural, and semantic information in a single framework. GAN makes them especially well-suited for IC tasks [7].
However, current methods also face challenges in balancing the reconstruction of both texture and structure despite significant progress in this field [8]. Demanding unique approaches for accomplishing structure and texture is the main challenge. Textures need consistency and fine-grained detail; structures must be cohesive and in line with the image geometry. Recent methods have started to use multi-stage models, like the suggested model. This model divides the structure completion task from the texture completion task in response to these difficulties [9]. In a GAN-based framework, the model can generate more visually appealing and semantically consistent completions by treating these elements separately.
This study introduced a new DL-based IC technique that makes use of a dual-stage GAN architecture. The first step in this study is to recreate the underlying structure of the missing region by using a sketch completion network [10]. The texture of the finished image was then refined using a texture completion network. This texture completion network is regarded as the second phase. In addition to improving the accuracy of recovering missing image components, this technique allows users flexibility to change structural elements and change the output as needed.
2. Related Work
For the purpose of addressing IC and Image Enhancement (IE), a DL framework was suggested by Chandak et al. [5]. This method utilizes the GAN method for effective image restoration. By using the Wasserstein Generative Adversarial Network (WGAN) architecture, missing regions in distorted images were filled. This GAN ensured the smoothly completed images. Through the implementation of the residual learning process with an enhanced network, these completed images were refined. Then, the quality of the completed images for CV applications was improved. 2.45% in Peak Signal Noise Ratio (PSNR) and 4% in Structural Similarity Index Measure (SSIM) were attained by the suggested method, and this suggested method overtakes the other current techniques in generating high-quality images.
In order to provide visually realistic content while maintaining local structures, Shamsolmoali et al. [11] devised deep feature-level semantic filtering. Two routes make up the suggested Dual-path Cooperative Filtering (DCF) model: one path produces semantically coherent reconstructions, while the other path extracts multi-level features and predicts dynamic kernels using Fast Fourier Convolution (FFC). The model showed better performance than State-Of-The-Art (SOTA) techniques on three difficult IC datasets. In IC tasks, it provides enhanced visual realism and structural coherence.
By merging Capsule Networks (CapsNets) with Globally and Locally (GL) consistent IC algorithms, Minglan et al. [12] suggested a multi-scale deep IC model called CapsNet-GL. The process begins with a randomly generated mask region on the original image, which is input into the complementary network for Feature Extraction (FE) and reconstruction using the GL algorithms. CapsNet enhances the global discriminator to extract richer multi-scale data, while local and global discriminators receive the complementation results from fusing local and global information. Experimental results on the CelebA-HQ dataset demonstrated that CapsNet-GL produces images with superior content and structure, efficiently extracting multi-scale visual information and achieving high accuracy in IC that aligns well with the original image.
Reddy et al. [13] suggested a CNN enhances the restoration of old photos by fixing multiple defects. CNNs, known for their superior ability to detect distinct shapes, patterns, and features in images, were employed as they are more effective than sequential multi-layer Neural Networks (NNs) in IP. The fusion of two latent branches in the network allows for a more comprehensive restoration process. By applying filters to every pixel of the image, this method achieves higher visual quality in comparison to SOTA restoration techniques, significantly improving the capacity to correct various imperfections in older photographs.
A novel StyleGAN-based IC network named Spectrum Hint Generative Adversarial Network (SH-GAN) was presented by Xu et al. [14]. This method incorporates a specialized spectral processing module named Spectrum Hint Unit (SHU). For the purpose of enhancing the compatibility with advanced DL methods, two new 2D spectral processing techniques Gaussian Split and Heterogeneous Filtering, were introduced, as it supports in resolving several image restoration challenges. Fréchet Inception Distance (FID) scores of 7.0277 and 3.4134 were attained, as it is executed on Flickr Faces High Quality (FFHQ) and Places2 benchmark datasets. These outcomes revealed that the suggested method surpasses other SOTA efficiency. The capability of the framework to resolve issues like pattern unawareness, fuzzy textures, and structural deformation was validated by the ablation studies. This increases the efficacy of the suggested method.
A DL-based IC technique was suggested by Xu et al. [15], as it emphasizes both the benefits and drawbacks of every method. Component optimization, network structure design optimization, and training method optimization are the 3 key areas that have been classified by the method. With the support of standard evaluation metrics and publicly available datasets, a comparative study was performed. Then, facial inpainting, general image correction, and object removal are the 3 classes of existing IC applications. In order to shed light on how IC is developing, the field's present difficulties, as well as potential future paths for development, were also examined.
min max x~p [log D(x)] + z∼p(z)[log(1 − D(G(z)))]. (1)
data
G Dax
A G is provided a noise vector, z, which is sampled from a normal distribution p(z). The G maps z to create x, a synthesized image complement.
Real-world data is predicted to be original by the first component of the equation ( x∼P [log D(x)]), which
data
is the log probability of the D.
As the log-likelihood of the discriminator, the second part of the equation ( z~p(z)[log(1 − D(G(Z)))], D, predicts the false information that G generates. Deep Convolutional Generative Adversarial Networks (DCGANs) was suggested by Radford et al. [23] as a simple way to train GAN for a variety of applications, including cross-domain IG networks and video data frame prediction.
For IG, Mirza and Osindero [24] developed Conditional Generative Adversarial Networks (CGANs) based on the availability of past information. CycleGANs have been employed recently for a variety of purposes, including training on unpaired image datasets, translating images, and obtaining good results in terms of accuracy, loss reduction, etc [25]. This work significantly enhances the performance of GANs in IG.
2.3 Architecture
Input images are mapped to the cartoon manifold using the generator network G. After training the model; cartoon stylization is generated. In order to spatially compress and encode the images, G starts with a flat Conv stage and then moves on to two down-Conv blocks. In this step, useful local signals are retrieved for transformation later on. To determine whether the input image is a real cartoon image, the D is utilized in conjunction with the generating network. Rather than using a full-image D, we employ a simple patch-level D with fewer parameters in D since determining if an image is cartoon or not is a less demanding task [26]. Cartoon style D is based on local aspects of the image, as opposed to object classification.
The network D is, therefore, intended to be shallow. Fig. 2 shows how the network uses two-stride Conv blocks to lower the resolution and encode crucial local features for classification after the flat layer stage.

3. Image Completion Generative Adversarial Network
A GAN framework consists of two CNNs. In order to deceive the D, G is first trained to produce output [27]. The other is the D, which establishes if the image is from the real target area or is a synthetic one. In order to accommodate the unique characteristics of cartoon images, develop the D and G networks.
3.1 Deep Network Architecture
3.1.1 Generator
The NN functions similarly to a decoder when the input vector's dimension is smaller than the output vector's dimension. Using the feature information, the face image is rebuilt, and the G, which functions as a decoder equivalent to the convolutional encoder E, receives as inputs the face feature vector z, the age vector a, and the gender vector g [7]. Due to the significant differences in the aging characteristics of the genders, the gender condition has been added. A face's aging synthesis is done based on its obvious gender, preventing gender from influencing the aging outcome. Fig. 3 shows the G network structure.

The merge vector comprising z, a, and g is the G input. According to the convolutional E structural diagram, the age data is an eight-dimensional one-hot vector, and z is a 60-dimensional z [28]. The gender data is a one-hot vector in two dimensions. The age and gender conditions will not significantly impact the generator if they are directly combined. To balance the effects of EigenVectors (EVs) and conditional vectors on the composite image, and is then copied seven times before merging to make a 56-dimensional vector, while g is copied thirty times to create a 60-dimensional vector.
Next, a 176-dimensional input is obtained by combining the EV z with the 116-dimensional vector that serves as the G conditional input. Fractional-strided Conv, which is sometimes referred to as deconvolution, is the most crucial network generation procedure. It is used in the micro-step Conv process [29]. A 5x5 Conv kernel with a 2×2 step size. The Output Layer (OL) of the generation network employs the Tanh Activation Function (AF), while all other layers use the Relu AF. Because Batch Normalisation (BN) is not used, this is comparable to the convolutional coding network.
3.1.2 Discriminator
In order to determine if an input image is a real face image or a synthetic one, the D must first distinguish between the two and then output a scalar value that represents the likelihood of the input image. Fig. 4 shows the D network structure. Fig. 2 illustrates this. The input is a 128 × 128 pixel RGB facial image (actual image or composite image), and the output is a scalar value between 0 and 1. The constraint is connected to the initial Convolution Layer (CL) in compliance with the condition GAN design guidelines [30].
In particular, a 16-pixel Feature Map (FM) is produced once the input image has gone through the first CL. This FM is then linked to the conditional FM following the extended copy to produce a 32-pixel FM. A scalar value that represents the likelihood is output once the conditional connection has been established effectively and the 32 FM have been entirely connected and convoluted.

3.2 Convolutional Neural Network
This network, which functions similarly to an artificial NN and has uses in _led such as IP and IR, is specifically utilized and built to handle pixel data [31]. The Input Layer (IL), OL, MultiLayer Perceptron (MLP) layer, Hidden Layer (HL), Fully Connected (FC) layers, and normal layers are among the numerous layers that make up CNN.
3.2.1 Loss function
Replicating a probability distribution is the objective of GAN. As a result, they demonstrate the discrepancy between the data distribution generated by the GAN and the real world using Loss Functions (LFs). Two LF, one for D training and the other for G training make up the majority of a GAN. One measurement of separation between the probability distributions is the source of both the D loss and the G loss. In any event, the G can only affect one term in the distance measure, which will affect how the fake data is distributed [32]. Because it will adapt to the distribution of the actual data, the other term is omitted during the G training. Even though the G loss and D loss come from the same formula, they seem to be distinct in the end. The D tries to expand the function, while the G tries to limit it: Ex[log(D(x))] + Ez[log(1 − D(G(z)))]D(x). D estimate that x is real. Ex: real-world instances.
4. Experimentation
To obtain reliable results, the following tests were carried out:
4.1 Image Completion Generation Using Generative Adversarial Network
Creating cartoon figures from scratch takes a lot of time, but with GAN, we can construct characters by using actual faces as input and processing them to produce high-density polygons. Using upscaling technology, the GAN model generates a high polygon density representation of the input image. When given enough CPUs and GPUs, a GAN model may produce a cartoon character from input in less time. For training, a GAN uses a collection of cartoon-style images (CartoonSet100K). The D network then receives the input. In order to create a fake image, we first take a dimensional noise vector and feed it into the G network.
Following that, the fake image is fed into the discriminator network, which compares it to the real image from the data set. If the two images are similar, the phony image is categorized as a predicted image; if not, it is sent back to the G network for retraining. This process, which is essentially highly time-consuming, keeps on until the fake image can deceive the D into thinking it is a real one. Therefore, we anticipate the image as a necessary output when the likelihood of correctly predicting the real image surpasses about 60% to 70%.
4.2 Generator
Utilizing the cartoonist application to handle user-taken real-world images. First, outlier data is eliminated, and the dataset is cleansed. Subsequently, the dataset is examined for any improper or Missing Values (MVs). After that, the dataset is run through the D and contrasted with the G random noise. The output is produced once the procedure achieves a high degree of precision.
5. Result
A GAN model, with sufficient CPUs and GPUs, may produce a cartoon character from input in less time. We take here input some different category images in Fig. 5.

The outcomes indicate that this technique outperforms the class strategies and can create outstanding animation images from real-world images with clear craftsman's styles, legible edges, and smooth hiding. The manual approach of obtaining labeled data takes a lot of time. Since GANs don't require labeled data, unlabelled data can be used to train them. GAN can be useful for converting real-world images into superior-quality images.
6. Conclusion
The outcomes show that our suggested GAN-based IC method effectively converts real-world images into finished, high-quality images. They guarantee sharper edges and enhanced structural integrity by implementing a novel edge-promoting adversarial LF. Furthermore, by including ℓ1 sparse regularisation into the VGG network's high-level FM for content loss, the model's adaptability is improved, allowing for more accurate texture replication and smoother shading. This approach also benefits from the ability to be trained on unlabeled data, making it highly adaptable to various applications. Overall, our method outperforms existing IC techniques, delivering superior results in both visual quality and efficiency.
References
- [1] Alkobi, N., Shaham, T. R., & Michaeli, T. (2023). Internal diverse image completion. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 648–658). IEEE. https://doi.org/10.1109/CVPRW59228.2023.00072
- [2] Chen, Q., Li, G., Xiao, Q., Xie, L., & Xiao, M. (2020). Image completion via transformation and structural constraints. EURASIP journal on image and video processing, 2020(44), 1–18. https://doi.org/10.1186/s13640-020-00533-3
- [3] Peng, X., Zhao, H., Wang, X., Zhang, Y., Li, Z., Zhang, Q., Wang, G., Peng, G., & Liang, H. (2023). C3N: content-constrained convolutional network for mural image completion. Neural computing and applications, 35(2), 1959–1970. https://doi.org/10.1007/s00521-022-07806-0
- [4] Wan, Z., Zhang, J., Chen, D., & Liao, J. (2021). High-fidelity pluralistic image completion with transformers. Proceedings of the IEEE/CVF international conference on computer vision (pp. 4692–4701). IEEE. https://doi.org/10.1109/ICCV48922.2021.00465
- [5] Chandak, V., Saxena, P., Pattanaik, M., & Kaushal, G. (2019). Semantic image completion and enhancement using deep learning. 2019 10th international conference on computing, communication and networking technologies (ICCCNT) (pp. 1–6). IEEE. https://doi.org/10.1109/ICCCNT45670.2019.8944750
- [6] Akimoto, N., Kasai, S., Hayashi, M., & Aoki, Y. (2019). 360-degree image completion by two-stage conditional gans. 2019 IEEE international conference on image processing (ICIP) (pp. 4704–4708). IEEE. https://doi.org/10.1109/ICIP.2019.8803435
- [7] Guo, J., & Liu, Y. (2019). Image completion using structure and texture GAN network. Neurocomputing, 360, 75–84. https://doi.org/10.1016/j.neucom.2019.06.010
- [8] Jiang, Y., Xu, J., Yang, B., Xu, J., & Zhu, J. (2020). Image inpainting based on generative adversarial networks. IEEE access, 8, 22884–22892. https://doi.org/10.1109/ACCESS.2020.2970169
- [9] Shao, C., Li, X., Li, F., & Zhou, Y. (2022). Large mask image completion with conditional GAN. Symmetry, 14(10), 2148. https://doi.org/10.3390/sym14102148
- [10] De Souza, V. L. T., Marques, B. A. D., Batagelo, H. C., & Gois, J. P. (2023). A review on generative adversarial networks for image generation. Computers & graphics, 114, 13–25. https://doi.org/10.1016/j.cag.2023.05.010
- [11] Shamsolmoali, P., Zareapoor, M., & Granger, E. (2023). Image completion via dual-path cooperative filtering. ICASSP 2023-2023 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 1–5). IEEE. https://doi.org/10.1109/ICASSP49357.2023.10097260
- [12] Minglan, Z., Weiqi, C., Yisheng, Z., Chun, Z., Linfu, S., & Min, H. (2023). A multi-scale deep image completion model fused capsule network. 2023 18th international conference on intelligent systems and knowledge engineering (ISKE) (pp. 288–293). IEEE. https://doi.org/10.1109/ISKE60036.2023.10481245
- [13] Reddy, B. S. K., SaiGnaneswar, J., Balaji, K., Reddy, K. S. V., & Reddy, K. L. (2023). Deep learning based image enhancing environment with noise suppression. 2023 second international conference on electronics and renewable systems (ICEARS) (pp. 1339–1344). IEEE. https://doi.org/10.1109/ICEARS56392.2023.10084977
- [14] Xu, X., Navasardyan, S., Tadevosyan, V., Sargsyan, A., Mu, Y., & Shi, H. (2023). Image completion with heterogeneously filtered spectral hints. Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 4591–4601). IEEE. https://doi.org/10.1109/WACV56688.2023.00457
- [15] Xu, Z., Zhang, X., Chen, W., Yao, M., Liu, J., Xu, T., & Wang, Z. (2023). A review of image inpainting methods based on deep learning. Applied sciences, 13(20), 11189. https://doi.org/10.3390/app132011189
- [16] Chen, D., Huang, T., Song, Z., Deng, S., & Jia, T. (2023). Agg-net: attention guided gated-convolutional network for depth image completion. Proceedings of the IEEE/CVF international conference on computer vision (pp. 8853–8862). IEEE. https://doi.org/10.1109/ICCV51070.2023.00813
- [17] Li, Y., & Yao, T. (2021). Image completion based on edge prediction and improved generator. Tehnički vjesnik, 28(5), 1590–1596. https://doi.org/10.17559/TV-20210616090311
- [18] Agrawal, P., & Kaushik, S. (2020). Image completion of highly noisy images using deep learning. Computational vision and bio-inspired computing: ICCVBIC 2019 (pp. 1031–1043). Springer, cham. https://doi.org/10.1007/978-3-030-37218-7_108
- [19] Talouki, A. G., Koochari, A., & Edalatpanah, S. A. (2024). Image completion based on segmentation using neutrosophic sets. Expert systems with applications, 238, 121769. https://doi.org/10.1016/j.eswa.2023.121769
- [20] Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018). Generative image inpainting with contextual attention. Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5505–5514). IEEE. https://doi.org/10.1109/CVPR.2018.00577
- [21] Teixeira, L., Oswald, M. R., Pollefeys, M., & Chli, M. (2020). Aerial single-view depth completion with image-guided uncertainty estimation. IEEE robotics and automation letters, 5(2), 1055–1062. https://doi.org/10.1109/LRA.2020.2967296
- [22] Yan, Z., Li, X., Li, M., Zuo, W., & Shan, S. (2018). Shift-net: image inpainting via deep feature rearrangement. Proceedings of the european conference on computer vision (ECCV) (pp. 1–17). Springer, cham. https://doi.org/10.1007/978-3-030-01264-9_1
- [23] Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. ArXiv preprint arxiv:1511.06434. https://doi.org/10.48550/arXiv.1511.06434
- [24] Mirza, M., & Osindero, S. (2014). Conditional generative adversarial nets. ArXiv preprint arxiv:1411.1784. https://doi.org/10.48550/arXiv.1411.1784
- [25] Zheng, C., Cham, T. J., & Cai, J. (2019). Pluralistic image completion. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1438–1447). IEEE. https://doi.org/10.1109/CVPR.2019.00153
- [26] Huang, C., & Yoshida, K. (2017). Evaluations of image completion algorithms: exemplar-based inpainting vs. Deep convolutional GAN. Deep convolutional gan. https://cs231n.stanford.edu/reports/2017/pdfs/306
- [27] Wu, X., Li, R. L., Zhang, F. L., Liu, J. C., Wang, J., Shamir, A., & Hu, S. M. (2019). Deep portrait image completion and extrapolation. IEEE transactions on image processing, 29, 2344–2355. https://doi.org/10.1109/TIP.2019.2945866
- [28] Ji, J., & Yang, G. (2019). Image completion with large or edge-missing areas. Algorithms, 13(1), 14. https://doi.org/10.3390/a13010014
- [29] Cai, J., Han, H., Shan, S., & Chen, X. (2019). FCSR-GAN: joint face completion and super-resolution via multi-task learning. IEEE transactions on biometrics, behavior, and identity science, 2(2), 109–121. https://doi.org/10.1109/TBIOM.2019.2951063
- [30] Guo, K., Cao, R., Kui, X., Ma, J., Kang, J., & Chi, T. (2019). LCC: towards efficient label completion and correction for supervised medical image learning in smart diagnosis. Journal of network and computer applications, 133, 51–59. https://doi.org/10.1016/j.jnca.2019.02.009
- [31] Egbert, A., Martone, A., Baylis, C., & Marks, R. J. (2020). Partial load-pull extrapolation using deep image completion. 2020 IEEE texas symposium on wireless and microwave circuits and systems (WMCS) (pp. 1–5). IEEE. https://doi.org/10.1109/WMCS49442.2020.9172302
- [32] Haselmann, M., Gruber, D. P., & Tabatabai, P. (2018). Anomaly detection using deep learning based image completion. 2018 17th IEEE international conference on machine learning and applications (ICMLA) (pp. 1237– 1242). IEEE. https://doi.org/10.1109/ICMLA.2018.00201