Application of computer vision and neural network algorithms to multisensor geodata for automated detection of finishing defects and geometric inconsistencies at a construction site

Abstract

The article examines the application of computer vision methods and neural network algorithms to multisensor geodata for automated detection of finishing defects and geometric inconsistencies at a construction site in the context of the industry's transition to Industry 4.0. The aim of the study is to develop and experimentally validate a comprehensive approach to the sensory fusion of terrestrial laser scanning data, photogrammetry, and BIM models, ensuring continuous and objective quality control without increasing inspectors' labor costs. The materials include high-density point clouds and more than 25,000 images collected at three residential and one industrial site, for which typical finishing defects and geometric deviations were expert-annotated; the methodological framework involves the use of a modified Mask R-CNN architecture for detecting and segmenting surface defects in images, a PointNet++ network for analyzing unstructured point clouds, ICP algorithms for aligning the as-built model with the BIM model, as well as experiments on sensor fusion of RGB, LiDAR, and contextual information. It is shown that the developed algorithmic system achieves precision and recall values for defect recognition with an F1-score above 0.9, with an average frame processing time of about 150 ms, while the automated assessment of geometric deviations demonstrates discrepancies with total station measurements of less than 0.3 mm, which is significantly below regulatory tolerances. Integration of RGB, LiDAR, and BIM leads to an increase in the F1-score to approximately 0.96 and a reduction in the probability of missed defects to less than 4%, with a statistically significant correlation identified between areas of surface unevenness and a higher frequency of cracks and delaminations. It is concluded that the proposed methodology exhibits high robustness to real construction site conditions and can serve as the basis for developing dynamic digital twins, proactive quality monitoring systems, and integration with augmented reality technologies and robotic platforms.