Visual Recognition and Prediction of Crack Propagation Trajectories

Abstract

Accurately detecting and tracking cracks on construction materials, such as concrete and rock surfaces, is of paramount importance in the field of structural engineering and materials science. The presence and propagation of cracks in these materials can have significant implications for structural integrity and safety, making it crucial to develop robust methods for crack detection and tracking. This work explores the application of computer vision and deep learning algorithms in the visual recognition and prediction of crack propagation trajectories in construction materials. Specifically, a Split Hopkinson Pressure Bar system and a static compression loading platform were constructed to simulate fracture processes in concrete and rock materials. The high-speed camera was used to capture the fracture process as a video or image sequence. An automatic pixel-level crack segmentation method was proposed to extract cracks at critical moments within the fracture video. Subsequently, the Long Short-Term Memory network was employed to learn temporal patterns and dependencies within the crack segmentation masks across consecutive frames. By leveraging the learned temporal patterns, the network can estimate how cracks will evolve and propagate in subsequent frames. The proposed method has potential applications in assessing structural integrity, understanding fracture mechanisms, and informing maintenance and repair decisions.

1. Introduction

Timely and accurate inspection and assessment of civil infrastructure, including buildings, roads, and underground tunnels, is crucial to ensure their continued safety and functionality. After years of operation, these structures may experience deterioration and damage, making it essential to identify and address potential issues promptly. Among the various types of distress, cracks are particularly significant as they can indicate severe structural integrity problems, posing risks to both the structure itself and the people in its vicinity. Therefore, the development of accurate, fast, and automated crack detection methods is of utmost importance in various civil engineering applications.

In recent decades, substantial advancements have been made in the automatic detection and quantification of cracks, thanks to the progress in hardware data acquisition systems. These advancements have enabled the implementation of sophisticated techniques for crack detection and analysis. By leveraging state-of-the-art sensors and imaging technologies, such as high-resolution cameras, LiDAR, or ground-penetrating radar (GPR), it is now possible to capture detailed information about the surface conditions of civil structures [1]. Automatic crack detection algorithms can utilize these acquired data to identify and locate cracks on structure surfaces. Computer vision techniques, including image processing and pattern recognition methods, play a vital role in analyzing the captured images or point cloud data and extracting relevant crack features [2, 3]. Machine learning algorithms, such as deep neural networks, can be trained on large datasets of annotated crack images to learn the characteristic patterns and textures associated with cracks. These models can then be deployed to accurately and efficiently detect cracks in real time or during post-processing.

Furthermore, advancements in data processing and computing capabilities have facilitated the development of efficient algorithms for crack quantification. Once cracks are detected, various parameters such as crack length, width, orientation, and depth can be measured to assess the severity and extent of the damage. These quantitative measurements are crucial for making informed decisions regarding maintenance, repair, or rehabilitation strategies.

The integration of remote sensing techniques, such as aerial or satellite imagery, with ground-based data acquisition systems has further enhanced the capabilities of crack detection and monitoring. By combining data from multiple sources, it becomes possible to obtain a comprehensive understanding of the structural health and performance of civil infrastructure over large areas.

In this study, we propose an automatic pixel-level crack segmentation method to extract cracks at critical moments within the fracture videos. This method leverages image processing techniques and deep learning models to identify crack regions accurately. By segmenting cracks at the pixel level, we can obtain detailed information about crack geometry, size, and orientation, which are essential for subsequent analysis and prediction. Furthermore, we employ a Long Short-Term Memory (LSTM) network, a type of recurrent neural network (RNN), to learn temporal patterns and dependencies within the crack segmentation masks across consecutive frames. The LSTM network is capable of capturing long-term dependencies and sequential information, making it suitable for modeling crack evolution over time. By leveraging the learned temporal patterns, the network can estimate how cracks will evolve and propagate in subsequent frames, enabling us to predict crack trajectories accurately.

The paper is structured as follows. Section 2 provides an overview of the related work on automatic crack detection and segmentation algorithms. Section 3 outlines the methodology employed in this study. Details regarding the experiments, specimens, and dataset preparation are presented in Section 4. Section 5 presents the results and discussions. Finally, Section 6 concludes the paper and discusses potential extensions of the work.

2. Related works

In the field of crack visual detection, significant research efforts have been devoted to developing effective methods for crack segmentation and analysis. This section presents a comprehensive review of the related work, focusing on four key categories: threshold-based, filter-based, machine learning, and deep learning methods.

Threshold-based methods involve setting a threshold value to distinguish crack and non-crack regions based on intensity or gradient information. Chen et al. [4] proposed a real-time image thresholding method for pavement crack detection. They calculated the mean and standard deviation of the pixel intensities in a gray-level pavement image to determine the threshold value. By comparing the pixel intensities with the threshold, cracks could be accurately segmented from the background. The method demonstrated the effectiveness of utilizing statistical measures for threshold determination in real-time crack detection scenarios.

Filter-based methods utilize various image filters to enhance crack features and improve segmentation results. These filters can be designed to capture specific crack characteristics such as texture or shape. Abdel-Qader et al. [5] conducted early research on visual detection of surface crack defects in bridges. They implemented and compared four crack detection image operator techniques, namely, Haar, Fast Harr Transform (FHT), Fast Fourier Transform (FFT), Sobel, and Canny, in the context of bridge monitoring and maintenance. The results indicated that FHT demonstrated higher reliability than the other three edge detection techniques in crack identification. Apart from above filters, Gabor filters offer selectivity for both scale and orientation, making them a valuable tool for crack detection. In numerous studies, researchers have utilized Gabor filters to detect cracks in images of pavements and bridges [6, 7].

Machine learning techniques have been widely applied to crack detection and segmentation tasks. These methods involve training a classifier or regression model on annotated crack images to automatically identify cracks. For example, the work in [8] explored the application of Support Vector Machines (SVM) for concrete crack detection. The training process involved feeding the SVM algorithm with a labeled dataset, where each sample was labeled as either “crack” or “non-crack.” The SVM algorithm learned to separate the two classes by finding an optimal hyperplane in a high-dimensional feature space. This hyperplane maximizes the margin between the two classes, allowing for accurate classification of new, unseen images. The results of the study demonstrated the feasibility and effectiveness of using SVM for concrete crack detection, achieving high detection accuracy and providing a reliable approach for crack identification in concrete structures. An example of employing a pavement crack detection framework based on random forests (RF) is presented in the work by Shi et al. [9]. They proposed a framework called CrackForest, specifically designed to identify complex cracks in pavement images. The CrackForest framework utilized random structured forests along with integral channel features to define crack tokens that incorporate structural information. These crack tokens represented crack regions within image patches, with each patch measuring 16x16 pixels and containing labeled crack edges. The CrackForest method demonstrated its capability to extract complex cracks with arbitrary topologies, including corners, curves, and lines, on the road crack dataset known as CFD. By leveraging the RF technique and incorporating integral channel features, the framework exhibited robust crack detection performance. It provided an effective approach for automated identification of complex cracks in pavement images, offering potential benefits in road inspection and maintenance. Kim et al. [10] proposed a machine learning-based method for concrete crack recognition, which aids in identifying the presence and location of cracks from surface images.

In recent years, with the increase in hardware computing power and data scale, deep learning algorithms have been widely applied to high-level feature learning in crack defect images. They have achieved remarkable results in the precise identification of complex crack defects in scenarios with complex background noise. Deep learning techniques, particularly convolutional neural networks (CNNs), have revolutionized crack detection and segmentation. These methods leverage the hierarchical representation learning capabilities of CNNs to automatically extract discriminative crack features. For instance, Zhang et al. [11] introduced a novel approach called CrackNet, which is based on Convolutional Neural Networks (CNN), for 3D asphalt pavement crack detection. In their studies, Zhang et al. compared the performance of CNN with other methods, including SVM and a 3D shadow modeling method. The results demonstrated the superiority of CNN over these alternative approaches. CNNs have the advantage of automatically learning discriminative features from raw data, allowing for more accurate and efficient crack detection in pavement images. Dung et al. [12] presented an algorithm for crack detection on concrete materials utilizing Fully Convolutional Networks (FCN). Their approach involved employing various CNN architectures, including VGG16, Inception-V3, and ResNet, as encoders to extract features. To evaluate the performance of the algorithm, Dung et al. conducted experiments on the Concrete-Crack-Ozgene dataset, which is a publicly available dataset commonly used for concrete crack analysis. The results of their study revealed that VGG16 outperformed Inception-V3 and ResNet in terms of crack image classification. Zhang et al. [13] proposed a novel context-aware deep convolutional semantic segmentation network for the effective detection of concrete surface cracks under various conditions.

3. Methodology

3.1 Overview

Figure 1 illustrates the framework overview of the visual recognition and prediction of crack propagation trajectories in this study. Firstly, we established static and dynamic loading experimental systems using a hydraulic machine and SHPB (Split Hopkinson Pressure Bar), respectively. Concrete, rock, and coal standard specimens were subjected to destructive tests at different loading rates. The crack propagation process of different materials under various loading conditions was recorded using a high-speed camera. Subsequently, the FPN (Feature Pyramid Network) was employed to identify and extract cracks, while ConvLSTM (Convolutional Long Short-Term Memory) was used for spatiotemporal modeling and analysis of the extracted cracks. Ultimately, predictive analysis of crack propagation in building materials was conducted.

Figure 1.

An overview of this work.

3.2 FPN for crack segmentation

Pixel-level crack segmentation plays a crucial role in accurately identifying and delineating crack regions within construction materials. To tackle this task, we used the Feature Pyramid Network (FPN), a popular architecture that has shown remarkable performance in various semantic segmentation tasks, including crack detection [14].

The FPN architecture is designed to address the challenge of effectively utilizing multi-scale features for pixel-level segmentation. It leverages a top-down pathway and lateral connections to create a feature pyramid with multiple levels of abstraction. This allows the network to capture both local and global contextual information, enabling accurate and precise segmentation. The FPN module consists of two main components: the top-down pathway and the lateral connections. The top-down pathway involves upsampling the higher-resolution feature maps to match the dimensions of the lower-resolution feature maps. This process allows the network to propagate fine-grained details from higher levels to lower levels, enhancing the ability to capture small-scale crack features. The lateral connections, on the other hand, involve the fusion of feature maps from different levels of the pyramid. This fusion operation helps to preserve both local and global contextual information by combining features at different scales. By integrating multi-scale features, the FPN can effectively handle cracks of varying sizes and complexities, improving the accuracy of pixel-level segmentation.

In this work, we employ ResNet as a pre-trained backbone network to extract high-level features from the input crack images. These features are then fed into the FPN module, which generates a set of feature maps at different resolutions, each corresponding to a specific scale of the input image. To perform pixel-level crack segmentation, we employ a segmentation head on top of the FPN module. This head typically consists of convolutional layers followed by upsampling and downsampling operations. The convolutional layers further refine the features obtained from the FPN, while the upsampling and downsampling operations adjust the resolution of the features to match the input image size and the desired output segmentation map.

During the training phase, we optimize the FPN network by minimizing the binary the Dice loss, and Intersection over Union (IoU) score. This loss function measures the discrepancy between the predicted crack segmentation map and the ground truth map at each pixel location. By backpropagating the error through the network, the FPN adjusts its parameters to improve the accuracy of crack segmentation.

The FPN-based pixel-level crack segmentation method offers several advantages. Firstly, the multi-scale feature fusion enables the network to capture crack details at different resolutions, improving the segmentation accuracy for cracks of varying sizes. Secondly, the top-down pathway facilitates the propagation of fine-grained details, enhancing the network’s ability to capture small-scale crack features. Lastly, the FPN architecture is flexible and can be easily integrated with different backbone networks, making it adaptable to various crack detection scenarios.

3.3 ConvLSTM for crack propagation prediction

The ConvLSTM [15] architecture extends the traditional LSTM by replacing fully connected layers with convolutional layers. This modification enables the network to process spatially structured data, such as images or video frames, directly. By integrating convolutional operations into the LSTM framework, the ConvLSTM can learn spatial patterns and temporal dependencies simultaneously. Specially, ConvLSTM layer consists of a grid of LSTM cells, where each cell maintains a hidden state and a cell state. The hidden state captures the learned representation of the crack segmentation masks, while the cell state retains the long-term memory of the temporal sequence. By processing each input frame sequentially, the ConvLSTM network can learn and update its hidden and cell states, capturing the evolution of cracks over time.

In other words, the CNN component enables the network to extract spatial features from crack segmentation masks, effectively capturing the local patterns and structural characteristics of cracks. This is particularly important when dealing with heterogeneous materials or complex crack networks. On the other hand, the LSTM component enables the network to learn long-term temporal dependencies and dynamics in crack evolution. By analyzing the sequential nature of crack segmentation masks, the LSTM can capture the temporal patterns and propagate information across frames, allowing the network to make accurate predictions about future crack configurations.

In this framework, the crack segmentation masks from consecutive frames are treated as input sequences to the ConvLSTM network. Each crack segmentation mask is first passed through a series of convolutional layers, which extract spatial features and encode local patterns. The output of the convolutional layers is then fed into the ConvLSTM layer, which models the temporal dynamics and dependencies between crack segmentation masks.

During the training phase, we optimize the ConvLSTM network by minimizing the prediction error between the predicted crack segmentation mask and the ground truth mask for each frame. We employ a suitable loss function, such as binary cross-entropy, to measure the discrepancy between the predicted and ground truth masks. By backpropagating the error through time, the network adjusts its parameters to improve the accuracy of crack propagation prediction. Once trained, the ConvLSTM network can be used for crack propagation prediction in unseen frames. Given the crack segmentation mask at time t, the network predicts the mask at time t + 1, representing the expected crack configuration in the subsequent frame. By iteratively applying this prediction process, we can forecast the crack propagation trajectories over multiple future frames.

In the subsequent sections, we will present the experimental setup, dataset acquisition, and training procedure in detail. We will also show the results to demonstrate the effectiveness and robustness of the ConvLSTM-based crack propagation prediction approach.

4. Experimental details

4.1 Experimental system

Figures 2 and 3 show the uniaxial compression loading and SHPB experimental setup used in this work, respectively. In Figure 2, the uniaxial compression loading experimental setup is depicted. This setup is typically used to apply a unidirectional compressive force to a specimen. It consists of a hydraulic machine that generates the compressive force and a load cell to measure the applied load. A manual servo-hydraulic machine with a maximum capacity of 80 MPa and a loading rate of 1.00 mm/s was used in the experiments. This machine allows for controlled application of compressive forces to the specimens. A high-resolution and high-frame-rate camera (GS3-U3-23S6M-C) was used in the experiment. It captures images at a resolution of 1920 × 1200 pixels and a frame rate of 163 fps. This camera effectively recorded the fracture behaviors of the specimens under uniaxial compressive loading.

Figure 2.

An illustration of the uniaxial compression loading system: (a) the schematic diagram; (b) the physical map.

Figure 3.

An illustration of the dynamic SHPB loading system.

Figure 3 illustrates the Split Hopkinson Pressure Bar (SHPB) experimental setup. The SHPB is commonly used to generate high-strain-rate loading conditions for dynamic material testing [16]. The experimental system is composed of four main components: SHPB bars, impact control system, stress pulse recorder, and video capture. To capture the images of the failure process, a full-color high-frame camera, specifically the Fastcam-SA5(16G) developed by PHOTRON corporation, was utilized. This camera is equipped with a Gigabit Ethernet high-speed communication interface, enabling it to capture images at a frame rate of up to 50,000 Frames Per Second (FPS). The camera has a resolution of 512 × 272 pixels, allowing for detailed and high-quality image capture of the crack propagation and failure process in the specimens.

4.2 Sample preparation

The study focused on conducting experiments on commonly used materials such as concrete, rock, and coal, as shown in Figure 4. To prepare the concrete specimens, cuboid shapes were used with approximate dimensions of 100 mm × 100 mm × 80 mm. In the case of concrete specimen preparation, a suitable concrete mix was formulated by combining cement, aggregates, and water in specific proportions. The mix was thoroughly blended to ensure uniformity and poured into molds designed to achieve the desired dimensions of approximately 100 mm × 100 mm × 80 mm. The specimens were then allowed to cure and harden under controlled conditions, ensuring proper hydration for the development of desired strength and properties.

Figure 4.

An illustration of the specimens: (a) concrete; (b) coal and rock.

Similarly, the rock and coal specimens were shaped into cylinders with a diameter of 50 mm and a height of 25 mm, following the recommendations of the International Society for Rock Mechanics (ISRM) for indirect tensile strength tests. The surfaces of the specimens were prepared to ensure flatness and smoothness, which are essential for accurate testing and analysis.

By adhering to the recommended dimensions specified by the ISRM for indirect tensile strength tests, the study ensured standardization and comparability of the specimens. This standardized sample preparation approach enables consistent and reliable evaluation of the mechanical properties and fracture behavior of concrete, rock, and coal materials.

4.3 Dataset preparation

In order to train the FPN (Feature Pyramid Network) network, a dataset was prepared by randomly selecting 30% of the data, which included 20 sets of crack propagation processes, consisting of nearly 300 images. The dataset was meticulously annotated to accurately label the crack propagation processes for different materials. Each frame in the dataset was carefully annotated, specifically focusing on annotating the cracks. Figures 5 and 6 illustrate the original images and extracted crack images of different materials under various loading conditions.

Figure 5.

An illustration of the original crack propagation data: (a) concrete material; (b) rock material; and coal material.

Figure 6.

An illustration of the annotated crack propagation data: (a) concrete material; (b) rock material; and coal material.

The primary objective of annotating this dataset was to create a comprehensive training set that would enable the FPN network to effectively learn and understand the crack propagation processes across various materials. Through meticulous manual annotation, professional annotators marked the cracks visible in each frame, ensuring precise identification and characterization of the cracks.

This meticulous annotation approach ensured the dataset’s high quality, providing valuable information for subsequent network training. By incorporating diverse crack propagation scenarios and materials, the dataset offers a robust and representative collection of crack images.

5. Results and discussion

5.1 Crack segmentation

Figure 7 illustrates the Dice loss and IoU score curves obtained during the training process of the FPN, as Figures 8–11 are the segmented crack images using trained FPN. The results demonstrate that the FPN network performs well in identifying and segmenting cracks in concrete, rock, and coal materials, under both static and dynamic loading conditions. These findings provide valuable data for the subsequent training of the ConvLSTM network.

Figure 7.

Training dice loss and IoU score of FPN model.

Figure 8.

An illustration of crack segmentation results of concrete specimen.

Figure 9.

An illustration of crack segmentation results of concrete specimen.

Figure 10.

An illustration of crack segmentation results of rock specimen.

Figure 11.

An illustration of crack segmentation results of coal specimen.

The Dice loss and IoU score curves indicate the network’s performance in terms of segmentation accuracy and overlap with ground truth annotations. The curves show consistent improvement during the training process, demonstrating the network’s ability to effectively capture the intricate details of cracks in different materials.

The segmented crack images reveal that the FPN network successfully identifies and delineates cracks in concrete, rock, and coal materials. The network demonstrates robust performance in recognizing cracks of varying shapes and sizes, enabling accurate segmentation. This capability is crucial for further analysis and characterization of crack propagation behavior. The FPN network, with its multi-scale feature fusion and top-down pathway, proves to be a powerful architecture for pixel-level crack segmentation in construction materials. By effectively utilizing contextual information and capturing crack details at different scales, the FPN enables accurate and precise identification of crack regions.

However, despite the overall success, there are some limitations observed in the crack segmentation results. For instance, in static loading scenarios, the network occasionally fails to accurately identify the full length of cracks, resulting in incomplete segmentation. Additionally, in dynamic loading cases, the network tends to overestimate the width of cracks, leading to slightly broader segmentation than the ground truth annotations. These limitations could be addressed through further fine-tuning of the network architecture and augmentation techniques or by incorporating additional training data. Future improvements in the network’s performance would enhance its ability to accurately segment cracks in both static and dynamic loading scenarios, facilitating more comprehensive analysis and prediction of crack behavior.

5.2 Crack propagation trajectories prediction

To accurately predict crack propagation trajectories, we employ a Convolutional Long Short-Term Memory (ConvLSTM) network. Based on experimental data and crack information extracted using the FPN network, we constructed a dataset comprising 217 sets of crack propagation videos. This dataset consisted of 73 sets of static loading-induced concrete crack videos, 74 sets of dynamic loading-induced rock crack videos, and 70 sets of dynamic loading-induced coal crack videos. Each video sequence included 10 frames depicting the progression of cracks. To train the ConvLSTM network, we utilized the first 5 frames as input data and the remaining 5 frames for crack propagation trajectory prediction.

Figure 12 illustrates the training and validation loss curves of the ConvLSTM network for both static and dynamic loading-induced crack propagation. These curves provide insights into the network’s learning process, showing the convergence of the training loss and indicating the network’s ability to capture the complex relationships between consecutive frames. Figures 13–15present the predicted crack propagation trajectories for different materials, showcasing the ConvLSTM network’s capabilities for crack trajectories generation. The network successfully predicts the paths along which cracks propagate in concrete, rock, and coal materials. This demonstrates its ability to capture the underlying patterns and tendencies of crack growth under various loading conditions.

Figure 12.

Training and validation loss of ConvLSTM model.

Figure 13.

An illustration of crack propagation trajectories prediction of concrete specimen under uniaxial compression loading.

Figure 14.

An illustration of crack propagation trajectories prediction of rock specimen under dynamic SHPB loading.

Figure 15.

An illustration of crack propagation trajectories prediction of coal specimen under dynamic SHPB loading.

However, alongside its achievements, the ConvLSTM network faces certain challenges and limitations. For example, inaccurate Prediction of Complex Crack Patterns: The ConvLSTM network may struggle to accurately predict crack propagation trajectories in cases where cracks exhibit complex and intricate patterns. This limitation may arise due to the network’s difficulty in capturing the subtle details and variations in such scenarios.

The accuracy of the LSTM network can also be evaluated qualitatively by visually inspecting the predicted crack masks and comparing them with the ground truth masks. If the predicted crack masks closely resemble the actual crack boundaries and capture the overall crack propagation trends, it indicates a higher level of accuracy in predicting crack propagation. However, it is important to note that crack propagation prediction is a challenging task due to the complex nature of crack behavior. Factors such as crack branching, merging, and irregular patterns can make accurate prediction difficult, especially in dynamic and complex fracture scenarios. To improve the accuracy of crack propagation prediction, several approaches can be considered: (1) High-quality Training Data: A large and diverse dataset of fracture videos with accurately annotated crack masks can enhance the training process and improve prediction accuracy. (2) Network Architecture Design: Optimizing the architecture of the LSTM network, such as adjusting the number of layers, hidden units, or incorporating attention mechanisms, can potentially enhance the network’s ability to capture temporal dependencies and improve prediction accuracy. (3) Data Augmentation: Applying data augmentation techniques, such as rotation, scaling, or adding noise, can help increase the robustness of the LSTM network and improve its generalization capability. (4) Ensemble Methods: Employing ensemble methods, such as combining predictions from multiple LSTM networks or incorporating other prediction models, can potentially improve the overall accuracy by leveraging diverse perspectives. It is worth noting that achieving high accuracy in crack propagation prediction is an ongoing research area, and the performance of the LSTM network can vary depending on the specific dataset and experimental conditions.

6. Conclusions

Accurate detection and tracking of cracks on construction materials are essential for understanding crack evolution and fracture mechanisms. By leveraging computer vision and deep learning techniques, we can develop automated systems that provide reliable crack detection, boundary tracking, and even predictive capabilities. These advancements contribute to enhancing structural integrity, safety, and durability in the field of construction materials.

This paper proposes the application of computer vision and deep learning techniques for the visual recognition and prediction of crack propagation trajectories in construction materials. By leveraging the power of these technologies, we can enhance crack analysis capabilities, improve structural health assessment, and enable proactive maintenance and repair strategies. The work combines experimental techniques, computer vision, and deep learning to advance crack detection and prediction in construction materials.

Further research can focus on enhancing the method’s robustness to complex crack patterns, improving generalization to unseen materials, and exploring the integration of additional contextual information to enhance crack prediction accuracy. The development of advanced network architectures and the utilization of larger and more diverse datasets can also contribute to improving the overall performance of the crack detection and prediction system.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 52004292), Shenzhen Polytechnic (Grant Nos. 6021310002K, 6022312015K and LHRC20220405), Shenzhen Science and Technology Program (ZDSYS20210929115800001).

Conflict of interest

The authors declare no conflict of interest.