Defect prediction method based on multi-feature parallel multi-stage neural network (MF-PMSNN)

[0001] This application claims priority to Chinese Patent Application No. 202410180822.6 with a filing date of Feb. 18, 2024. The content of the aforementioned application, including any intervening amendments thereto, is incorporated herein by reference.

[0002] The present disclosure relates to the technical field of learning prediction, and specifically, to a defect prediction method based on a multi-feature parallel multi-stage neural network (MF-PMSNN).

[0003] A deep learning framework for defect prediction in additive manufacturing (AM) based on thermal imaging-based in-situ monitoring measures surface radiance during laser exposure through short-wave infrared thermal imaging, and reconstructs a thermal history of a component. This framework consists of a data preprocessing workflow and a supervised deep learning classifier architecture. A custom convolutional neural network model is used for classification, training, and testing, and a trajectory dataset comes from the thermal imaging-based in-situ monitoring in manufacturing of a stainless steel test component. Research has shown that this framework has high accuracy in predicting porosity of a small-volume keyhole.

[0004] In the prior art, AM processing without using deep learning usually depends on a traditional image processing technique, and its detection is limited by prior knowledge and a manually designed feature. Traditional methods have poor adaptability to a complex process condition and a material change, making it difficult to deal with an environmental change in AM. These methods require a manually designed rule or feature extractor. Different from the deep learning, these methods cannot perform autonomous learning based on data, thus limiting a system in obtaining and applying new knowledge. Therefore, a traditional method lacking real-time feedback may not be able to give full play to its detection and prediction capabilities in a complex AM platform.

[0005] The AM involves complex fusion and pore formation, while a deep learning model can automatically learn and extract complex features such as a molten pool status and porosity without a need to manually design a feature extractor. With powerful capabilities for processing high-dimensional and large-scale data, the deep learning model can adapt to multimodal data in AM processing, including thermal imaging and X-ray computed tomography (XCT). A good deep learning model usually has a strong generalization capability and can accurately predict defects under a new process condition and material. Therefore, it is an effective method to use the deep learning for defect prediction in the AM processing.

[0006] The deep learning framework for the defect prediction based on the thermal imaging-based in-situ monitoring is an effective method in a current AM process. Although the thermal imaging-based in-situ monitoring is a most commonly used workpiece status monitoring method in the AM, the thermal imaging-based in-situ monitoring monitors only a condition of a surface layer, which may result in distorted deep learning prediction. In the AM, laser acts on metal powder of the surface layer, which may affect molten pool statuses and porosity of lower layers. Thermal imaging-based monitoring only covers the surface layer, and data is not comprehensive, resulting in inaccurate deep learning prediction. Therefore, it is necessary to combine the multimodal data and use the deep learning model to capture workpiece information more comprehensively, thereby improving accuracy of the defect prediction.

[0007] In view of the mentioned shortcomings in the prior art, the present disclosure provides a defect prediction method based on an MF-PMSNN.

[0008] In order to achieve the above effect, the present disclosure adopts the following technical solutions.

[0009] The present disclosure provides a defect prediction method based on an MF-PMSNN, including the following steps:

• • [0010] step 1, obtaining a trajectory dataset, and preprocessing data of a defect of a workpiece in AM;
  • [0011] step 2, building an MF-PMSNN, where the MF-PMSNN has N branch networks, and each of the branch networks includes a Conv-Stem layer, a Stage X module, an average pooling layer, a fully connected layer, and a classifier that are sequentially connected, capturing space and timing sequence information in multimodal data, and obtaining a molten pool status and a porosity distribution in the data, where specific steps are as follows:
  • [0012] step 2.1, inputting preprocessed data into the Conv-Stem layer for feature extraction to obtain an eigenvalue;
  • [0013] step 2.2, inputting the eigenvalue into the Stage X module for data processing, where the Stage X module includes a pooling layer and a local-to-global feature extraction module that are sequentially connected;
  • [0014] step 2.3, inputting the eigenvalue processed by the Stage X module into the average pooling layer for average pooling, and extracting a feature of the processed eigenvalue; and
  • [0015] step 2.4, inputting the feature of the processed eigenvalue into the fully connected layer, constructing an output layer through the fully connected layer, using an activation function to map the feature inputted into the fully connected layer, and inputting mapped feature into the classifier for defect category recognition to obtain a defect category label;
  • [0016] step 3, evaluating an output classification result based on evaluation indicators; and
  • [0017] step 4, performing real-time defect prediction, and deploying a trained MF-PMSNN model to actual production of the workpiece.

[0018] In one embodiment, the obtaining a trajectory dataset in the step 1 includes:

• • [0019] collecting trajectory data by using thermal imaging-based in-situ monitoring and XCT-based in-situ monitoring devices, wherein the trajectory data comprises thermal imaging-based in-situ monitoring data and XCT-based in-situ monitoring data;
  • [0020] defining a label related to the defect of the workpiece through supervised learning, wherein the label includes the molten pool status and a porosity distribution; and
  • [0021] effectively matching the thermal imaging-based in-situ monitoring data and the XCT-based in-situ monitoring data to ensure temporal and spatial consistency between the thermal imaging-based in-situ monitoring data and the XCT-based in-situ monitoring data.

[0022] In one embodiment, the preprocessing the data of the defect of the workpiece in AM in the step 1 includes:

• • [0023] step 1.1, obtaining the trajectory dataset;
  • [0024] step 1.2, performing data cleaning to clean a missing value, an outlier, or an incorrect label in the trajectory dataset;
  • [0025] step 1.3, standardizing and normalizing the data;
  • [0026] step 1.4, labeling a defect in the trajectory dataset and a category to which each sample in the trajectory dataset belongs;
  • [0027] step 1.5, performing data enhancement on the trajectory dataset; and
  • [0028] step 1.6, dividing the trajectory dataset into a training set, a validation set, and a test set.

[0029] In one embodiment, the trajectory dataset in the step 1.1 includes:

• • [0030] a model of a processing engine bed, ambient humidity, an ambient temperature, a molten pool length, a molten pool width, a molten pool eccentricity ratio, an average molten pool temperature, and a maximum molten pool temperature in the AM.

[0031] In one embodiment, the inputting the preprocessed data into the Conv-Stem layer for feature extraction in the step 2.1 includes:

• • [0032] performing 3×3 convolution with a stride of 2 on the processed data to halve a number of an output eigenvalue, performing batch normalization, and using an activation function Gaussian Error Linear Unit (GELU) in network training:

[0033] $\mathrm{GELU}\left(x\right)=0.5x\left(1+\tanh \left(\sqrt{\frac{2}{x}}\left(x+0.044715x^3\right)\right)\right).$

[0034] In one embodiment, the data processing in the step 2.2 includes pooling the eigenvalue by the Stage X module which specifically comprises:

• • [0035] pooling the eigenvalue outputted from the Conv-Stem layer, and capturing local information and global information of the eigenvalue, where the local-to-global feature extraction module includes a contextual perception block, a first batch normalization layer, a lightweight multi-head self-attention mechanism, a second batch normalization layer, and a multi-layer perceptron that are sequentially connected;
  • [0036] enhancing the eigenvalue by using the contextual perception block to obtain enhanced data, performing 3×3 depth-separable convolution on the enhanced data, and inputting obtained data into the first batch normalization layer, performing layer normalization and calculating a statistical magnitude for each input data sample by the first batch normalization layer, thereby facilitating convergence and stability of the model; and
  • [0037] executing the lightweight multi-head self-attention mechanism to capture a long distance dependency in data inputted into the lightweight multi-head self-attention mechanism; and processing the long distance dependency by using the second batch normalization layer and the multi-layer perceptron, to obtain an output vector.

[0038] In one embodiment, the constructing the output layer through the fully connected layer in the step 2.4 includes:

• • [0039] connecting each input neuron of the fully connected layer to each output neuron to form a fully connected network, and converting the mapped feature into a final output, where an expression of the fully connected layer is as follows:

[0040] $\mathrm{output}=\mathrm{activation}\left(\mathrm{input}\times \mathrm{weights}+\mathrm{biases}\right)$
where input represents the data inputted into the fully connected layer, weights represents a weight matrix, biases represents a bias, and activation represents the activation function.

[0041] The method further includes the following step after step 3: setting a loss function, using an Adam optimizer to optimize the MF-PMSNN, and obtaining an optimal MF-PMSNN when the loss function has a minimum value.

[0042] In one embodiment, the method further includes:

• • [0043] feedback and continuous improvement: collecting feedback information from a user and a professional on an evaluation result of the model after testing, adjusting the evaluation indicator, a model parameter, and a training method based on the feedback information, and regularly updating the trajectory dataset.

[0044] Compared with the prior art, the technical solutions of the present disclosure have the following beneficial effects:

[0045] The present disclosure combines and effectively matches thermal imaging-based in-situ monitoring data and XCT-based in-situ monitoring data to ensure temporal and spatial consistency between the thermal imaging-based in-situ monitoring data and the XCT-based in-situ monitoring data. In this way, a molten pool status and a pore of a workpiece can be captured more comprehensively. An MF-PMSNN is proposed to obtain a molten pool status and a porosity distribution in the data and perform defect prediction. The present disclosure solves a problem of distorted deep learning prediction caused by using only the thermal imaging-based in-situ monitoring data, making the defect prediction more accurate.

[0049] The implementations of the present disclosure are described below with reference to the accompanying drawings and preferred embodiments, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. The present disclosure can also be implemented or applied through other different specific implementations. Based on different viewpoints and applications, various modifications or amendments can be made to various details of this specification without departing from the spirit of the present disclosure. It should be understood that the following preferred embodiments are only used for illustrating the present disclosure, rather than limiting the protection scope of the present disclosure.

[0050] It should be noted that the drawings provided in the following embodiments merely illustrate the basic concepts of the present disclosure schematically. Therefore, the drawings only show components related to the present disclosure rather than being drawn according to quantities, shapes, and sizes of components in actual implementation, patterns, quantities, and proportions of components in actual implementation may be changed randomly, and a component layout may be more complex.

[0051] Before the embodiments of the present disclosure are described, relevant terms involved in the embodiments of the present disclosure are explained as follows:

[0052] MF-PMSNN: It is short for a multi-feature parallel multi-stage neural network.

[0053] AM: As a manufacturing technique, the AM constructs a three-dimensional object by stacking or adding a material layer by layer, and is opposed to a traditional manufacturing method in which a material is cut or removed.

[0054] Molten pool: The molten pool is a local melting region of metal powder due to laser irradiation, and is a start point of constructing a new layer.

[0055] Porosity: The porosity is a percentage of a pore or a bubble inside a component, which has a significant impact on strength and quality of the component.

[0056] F1-Score: F1-Score is an indicator for evaluating the overall function of the prediction model. It is a harmonic mean of the precision and the recall rate. F1-Score is especially suitable for applying in the cases with unbalanced dataset. A scope of F1-Score is [0, 1], and when F1-Score is about 1, it represents that the prediction model is well balanced between the precision and the recall rate and the prediction model is capable of recognizing the samples with defect effectively and error classification to the samples can be reduced significantly.

[0057] This embodiment provides a defect prediction method based on an MF-PMSNN. Referring to FIG. 1 to FIG. 3, the following steps are included.

[0058] In step 1, a trajectory dataset is obtained, and data of a workpiece defect in AM is preprocessed.

[0059] In step 2, an MF-PMSNN is built. The MF-PMSNN has N branch networks, and each of the branch networks includes a Conv-Stem layer, a Stage X module, an average pooling layer, a fully connected layer, and a classifier that are sequentially connected. Space and timing sequence information in multimodal data is captured, and a molten pool status and a porosity distribution in the data are obtained. Specific steps are as follows:

[0060] In step 2.1, preprocessed data is input into the Conv-Stem layer for feature extraction to obtain an eigenvalue.

[0061] In step 2.2, the eigenvalue is input into the Stage X module for data processing. The Stage X module includes a pooling layer and a local-to-global feature extraction module that are sequentially connected.

[0062] In step 2.3, the eigenvalue processed by the Stage X module is input into the average pooling layer for average pooling, and a feature of the processed eigenvalue is extracted.

[0063] In step 2.4, the feature of the processed eigenvalue is input into the fully connected layer, and an output layer is constructed through the fully connected layer. An activation function is used to map the feature inputted into the fully connected layer, and mapped feature is input into the classifier for defect category recognition to obtain a defect category label. In other words, output data is mapped onto a predefined category through the classifier.

[0064] In step 3, an output classification result is evaluated based on evaluation indicators.

[0065] In step 4, real-time defect prediction is performed. A trained MF-PMSNN model is deployed to actual production of the workpiece to receive and process multimodal data in a timely manner, including thermal imaging data and XCT data.

[0066] The present disclosure recognizes key information such as the molten pool status and porosity of the workpiece in real time through model inference. An anomaly detection mechanism is introduced to effectively handle an output anomaly. Each real-time prediction result is recorded, and a production log is established to support subsequent analysis and optimization. This real-time defect prediction strategy helps to detect and handle a defect in AM processing in a timely manner, minimize a scrap rate, and improve production efficiency and product quality. Through continuous monitoring and feedback, the defect of the workpiece in the AM processing can be accurately predicted.

[0067] In the present disclosure, model training and optimization ensure that a format of a multimodal trajectory dataset matches an MF-PMSNN model. An appropriate loss function and optimizer are selected, and stability is improved through learning rate adjustment, batch normalization, regularization, and other techniques. This neural network model can train a plurality of features simultaneously and set independent parameters to train different features. After the training, the model can output training results in parallel. A training set is used to train the model, and a validation set is used to perform validation and handle overfitting or underfitting. With the help of a feedback mechanism, the model is adjusted and optimized based on the validation set and a feedback of an actual application, which may include adding training data and adjusting a model structure. A model weight is regularly saved, and accuracy of the neural network model in predicting defects under different process conditions is improved through a plurality of iterations, thereby enhancing generalization and application value. A loss function is selected, such as a mean square error or a cross entropy. The model is trained by using a backpropagation algorithm and the optimizer, and hyperparameter adjustment is performed on the validation set to optimize performance. The model is trained by using a registered trajectory dataset and optimized by using the feedback mechanism. This ensures a generalization capability of the model to different process conditions and materials, thereby improving accuracy and robustness of defect prediction.

[0068] As a preferred technical solution, in one embodiment, the obtaining the trajectory dataset in the step 1 includes following operations:

[0069] A format of the input data is matched with a design of the model. Trajectory data is collected by using thermal imaging-based in-situ monitoring and XCT-based in-situ monitoring devices, including thermal imaging-based in-situ monitoring data and XCT-based in-situ monitoring data. More comprehensive information of the workpiece is provided.

[0070] A label related to the defect of the workpiece is defined through supervised learning. The label includes the molten pool status and a porosity distribution.

[0071] The thermal imaging-based in-situ monitoring data and the XCT-based in-situ monitoring data are effectively matched to ensure temporal and spatial consistency between the thermal imaging-based in-situ monitoring data and the XCT-based in-situ monitoring data. This solves a problem of distorted deep learning prediction caused by only using surface layer information.

[0072] As a preferred technical solution, in this embodiment, the preprocessing data of a defect of a workpiece in AM in the step 1 includes the following steps:

[0073] In step 1.1, the trajectory dataset is obtained.

[0074] In step 1.2, data cleaning is performed to clean a missing value, an outlier, or an incorrect label in the trajectory dataset.

[0075] In step 1.3, the data is standardized and normalized.

[0076] In step 1.4, a defect in the trajectory dataset and a category to which each sample in the trajectory dataset belongs are labeled.

[0077] In step 1.5, data enhancement is performed on the trajectory dataset to generate more training samples.

[0078] In step 1.6, the trajectory dataset is divided into the training set, the validation set, and a test set, and complete input data is constructed.

[0079] Multimodal data fusion aims to integrate thermal imaging and XCT sensor data to obtain comprehensive information of the workpiece. A high-quality device is selected to ensure that its performance meets a real-time and high-resolution requirement. An optimal installation location is determined to enable the device to cover an entire processing region. Synchronous data collection is maintained, and a real-time transmission mechanism is established to support the defect prediction. The device is maintained and calibrated regularly to ensure performance and data accuracy. Data registration ensures the temporal and spatial consistency between the thermal imaging-based in-situ monitoring data and the XCT-based in-situ monitoring data. Specifically, a key feature point, a region, and a descriptor is extracted, and preliminary registration is performed by using a matching method. A timing sequence relationship between component shape change/deformation and data is considered to ensure continuity. A registration result is regularly evaluated to ensure accuracy and stability. A real-time feedback mechanism is established to monitor and adjust the registration to adapt to a production condition.

[0080] As a preferred technical solution, in this embodiment, the trajectory dataset in the step 1.1 includes:

• • [0081] a model of a processing engine bed, ambient humidity, an ambient temperature, a molten pool length, a molten pool width, a molten pool eccentricity ratio, a molten pool perimeter, an average molten pool temperature, a maximum molten pool temperature, time when a threshold 1200 K is exceeded, time when a threshold 1680 K is exceeded, and time when a threshold 2400 K is exceeded in the AM.

[0082] As a preferred technical solution, in this embodiment, the inputting the preprocessed data into the Conv-Stem layer for feature extraction in the step 2.1 includes the following operations:

[0083] 3×3 convolution with a stride of 2 is performed on the preprocessed data to halve the number of the output eigenvalue, and batch normalization is performed. An activation function GELU is used in network training:

[0084] $\mathrm{GELU}\left(x\right)=0.5x\left(1+\tanh \left(\sqrt{\frac{2}{x}}\left(x+0.044715x^3\right)\right)\right).$

[0085] Network training is accelerated and robustness is improved. The 3×3 convolution is performed for further feature extraction and combination, which helps to extract and learn a higher-level feature representation.

[0086] In the present disclosure, in order to maintain spatial invariance, a space size remains unchanged during channel transformation, and the activation function GELU is used to perform nonlinear activation on a channel of a target image feature. This ensures randomness in a channel mapping process and makes training parameters of the neural network model in the present disclosure more robust.

[0087] As a preferred technical solution, in one embodiment, the data processing in the step 2.2 includes pooling the eigenvalue by the Stage X module which specifically the following operations:

[0088] The eigenvalue outputted from the Conv-Stem layer is pooled as the input data, and a plurality of convolution operations are performed during the pooling to enable the network to learn a more complex feature. Local information and global information of the input data are effectively captured in a modular structure that gradually extracts a global feature from a local feature. The local-to-global feature extraction module includes a contextual perception block, a first batch normalization layer, a lightweight multi-head self-attention mechanism, a second batch normalization layer, and a multi-layer perceptron that are sequentially connected.

[0089] The eigenvalue is enhanced by using the contextual perception block to obtain enhanced data, 3×3 depth-separable convolution is performed on the enhanced data, and obtained data is input into the first batch normalization layer to reduce a parameter quantity of a feature set at a specific level in a lightweight model. The first batch normalization layer performs layer normalization and calculates a statistical magnitude for each input data sample, facilitating convergence and stability of the model.

[0090] The lightweight multi-head self-attention mechanism is executed to capture a long distance dependency in the data inputted into the lightweight multi-head self-attention mechanism. the long distance dependency is processed by using the second batch normalization layer and the multi-layer perceptron, to obtain an output vector. A nonlinear modeling capability of a nonlinear mapping enhancement model is introduced. The entire network includes a plurality of modules to build a powerful end-to-end model. The multi-layer perceptron includes a plurality of fully connected layers. A multi-head attention mechanism is introduced to obtain more complete feature information and improve the generalization capability of the network.

[0091] As a preferred technical solution, in one embodiment, the constructing the output layer through the fully connected layer in the step 2.4 includes the following operations:

[0092] Each input neuron of the fully connected layer is connected to each output neuron to form a fully connected network, and the mapped feature is converted into a final output. An expression of the fully connected layer is as follows:

[0093] $\mathrm{output}=\mathrm{activation}\left(\mathrm{input}\times \mathrm{weights}+\mathrm{biases}\right).$

[0094] In the above expression, input represents data inputted into the fully connected layer, weights represents a weight matrix, biases represents a bias, and activation represents the activation function. The activation function is used to introduce nonlinearity. The average pooling layer is used for pooling dimensionality reduction to extract a potential feature from the input data.

[0095] As a preferred technical solution, in one embodiment, the evaluating an experimental result based on evaluation indicators in the step 3 is as follows:

[0096] The evaluation indicators include Top-k accuracy, precision, a recall rate, and an F1-Score.

[0097] When k=1, a formula for the Top-k accuracy is as follows:

[0098] $acc=\frac{TP+TN}{TP+TN+FP+FN}$

[0099] A formula for the precision is as follows:

[0100] $\mathrm{pre}=\frac{TP}{\left(TP+FP\right)}$

[0101] A formula for the recall rate is as follows:

[0102] $\mathrm{recall}=\frac{TP}{\left(TP+FN\right)}$

[0103] A formula for the F1-Score is as follows:

[0104] $F1-\mathrm{Score}=\frac{2\times \mathrm{pre}\times \mathrm{recall}}{p\mathrm{re}+\mathrm{recall}}=\frac{2TP}{2TP+FP+FN}$
where TP represents an actual number of samples with defect, TN represents an actual number of samples without defect, FP represents a predicted number of samples with defect, and FN represents a predicted number of samples without defect.

[0105] As a preferred technical solution, in this embodiment, after the step 3, the method further includes: setting the loss function, using an Adam optimizer to optimize the MF-PMSNN, and obtaining an optimal MF-PMSNN when the loss function has a minimum value. In this way, a generalization capability of the model is improved. The loss function is used to evaluate a degree to which a predicted value differs from a true value. The loss function may be a cross entropy loss or a mean square error loss.

[0106] As a preferred technical solution, in this embodiment, the method includes:

• • [0107] feedback and continuous improvement: collecting feedback information from a user and a professional on an evaluation result of the model after testing, adjusting the evaluation indicator, a model parameter, and a training method based on the feedback information, and regularly updating the trajectory dataset.

[0108] It should be noted that an internal structure of an object can be visualized according to an X-ray attenuation principle. The X-ray attenuation principle can provide basic information of a currently detected object for diagnosis and preservation. XCT is an advanced non-destructive testing technique that can visually display the detected object in a form of a two-dimensional or three-dimensional digital image without causing damage to the detected object. The XCT is based on a large number of different projections, which are then used to reconstruct a cross-sectional image (or slice) of the detected object. Each cross-sectional image is a digital image composed of a pixel matrix, and each pixel in the image represents a volume element (voxel) of an object. Based on the XCT technology, the embodiments of the present disclosure can be implemented with high precision, and the following beneficial effects are achieved:

[0109] In order to solve a problem that a thermal imaging-based in-situ monitoring method can only monitor a status of a surface layer, the present disclosure proposes an MF-PMSNN. This framework combines thermal imaging-based in-situ monitoring data and XCT-based in-situ monitoring data, and accurately registers these two types of trajectory datasets to more comprehensively capture a molten pool status and a pore of a workpiece. This solves a problem of distorted deep learning prediction caused by only using the thermal imaging-based in-situ monitoring data, making defect prediction more accurate.

[0110] Based on a deep learning framework for defect prediction, the present disclosure analyzes a lack of a real-time defect feedback and adjustment capability in an AM processing platform when the defect prediction is performed without using a deep learning method, as well as a problem of a single input trajectory dataset of the deep learning framework because only one type of thermal imaging sensor is used when the deep learning method is used for prediction. Combined with a deep learning method for defect prediction based on XCT-based in-situ monitoring, for a problem of an incomplete deep learning framework for the defect prediction due to insufficient layers of collected workpiece data, the present disclosure proposes a deep learning framework that uses a trajectory dataset obtained by accurately registering the thermal imaging-based in-situ monitoring data and XCT reference data as input data. This solves a problem of inaccurate defect prediction, and improves processing accuracy of the AM method. This method is improved on a basis of the thermal imaging-based in-situ monitoring to obtain an effective deep learning framework.

[0111] It is apparent that the above embodiments are merely intended to describe the present disclosure clearly, rather than to limit the implementations of the present disclosure. The person of ordinary skill in the art may make modifications or variations in other forms based on the above description. There are no need and no way to exhaust all the implementations. Any modification, equivalent substitution, and improvement made within the spirit and principle of the present disclosure should fall within the protection scope of the claims of the present disclosure.