Welding system, welding method, welding support device, program, learning device, and method of generating trained model

[0001] The present invention relates a welding system, a welding method, a welding support device, a program, a learning device, and a method of generating a trained model.

[0002] PTL 1 discloses a technique in which images of a molten pool, and a ripple shape and a geometric shape of a fillet portion are captured by a camera, and received by a processor which communicates with a database that stores associated potential defects of welding along with an image of a molten pool of a mock weld zone and an image of a ripple shape of a mock weld zone and a geometric shape of a fillet portion, and calculates an aggregate probability that a defect occurs at a welding position corresponding to an image captured by a camera based on the potential defects associated in the database.

• [0003] PTL 1: Japanese Unexamined Patent Application Publication No. 2017-106908

[0004] Meanwhile, some welding defects occur in a surface part, whereas other welding defects occur internally, and welding defects which can be inspected with a camera are mostly limited to the defects that occur in a surface part. In addition, in arc welding, a weld zone may be interfered by arc light, thus it is difficult to inspect the weld quality with high accuracy only by images obtained by a camera.

[0005] In addition, welding sound and the waveform of the current or the voltage of a welding power source are disturbed by a welding failure, but are not necessarily associated with an occurrence of a welding defect, thus it is difficult to inspect the weld quality with high accuracy only with measured data of each amount.

[0006] The present invention has been made in consideration of the above-mentioned problem, and it is a main object to provide a welding system, a welding method, a welding support device, a program, a learning device, and a method of generating a trained model that are capable of improving the accuracy of prediction of a welding quality.

[0007] In order to solve the above-mentioned problem, a welding system in an aspect of the present invention includes: a welding device; a plurality of sensors in different types, which detect an event according to welding performed by the welding device; and an estimation unit that estimates an abnormality of the welding performed by the welding device from a plurality of pieces of detection data generated by the plurality of sensors, using a trained model that is pre-generated by machine-learning by taking, as input data, a plurality of pieces of data for learning obtained by detecting events according to welding by same types of sensors as the plurality of sensors, and, as training data, labels representing whether the welding is normal or abnormal.

[0008] In addition, a welding method in another aspect of the present invention includes: detecting an event according to welding performed by a welding device by a plurality of sensors in different types; and estimating an abnormality of the welding performed by the welding device from a plurality of pieces of detection data generated by the plurality of sensors, using a trained model that is pre-generated by machine-learning by taking, as input data, a plurality of pieces of data for learning obtained by detecting events according to welding by same types of sensors as the plurality of sensors, and, as training data, labels representing whether the welding is normal or abnormal.

[0009] In addition, a welding support device in another aspect of the present invention includes: an acquisition unit that acquires a plurality of pieces of detection data generated by a plurality of sensors in different types, which detect an event according to welding performed by a welding device; and an estimation unit that estimates an abnormality of the welding performed by the welding device from the plurality of pieces of detection data generated by the plurality of sensors, using a trained model that is pre-generated by machine-learning by taking, as input data, a plurality of pieces of data for learning obtained by detecting events according to welding by same types of sensors as the plurality of sensors, and, as training data, labels representing whether the welding is normal or abnormal.

[0010] In addition, a program in another aspect of the present invention causes a computer to execute a process including: acquiring a plurality of pieces of detection data generated by a plurality of sensors in different types, which detect an event according to welding performed by a welding device; and estimating an abnormality of the welding performed by the welding device from the plurality of pieces of detection data generated by the plurality of sensors, using a trained model that is pre-generated by machine-learning by taking, as input data, a plurality of pieces of data for learning obtained by detecting events according to welding by same types of sensors as the plurality of sensors, and, as training data, labels representing whether the welding is normal or abnormal.

[0011] In addition, a learning device in another aspect of the present invention includes: an acquisition unit that acquires a data set for learning including a plurality of pieces of data for learning obtained by detecting events according to welding by a plurality of sensors in different types, and labels representing whether the welding is normal or abnormal; and a learning unit that generates a trained model to estimate an abnormality of welding from a plurality of pieces of detection data obtained by detecting events according to welding with same types of sensors as the plurality of sensors, by taking the plurality of pieces of data for learning as input data, and the labels as training data.

[0012] A method of generating a trained model in another aspect of the present invention includes: acquiring a data set for learning including a plurality of pieces of data for learning obtained by detecting events according to welding by a plurality of sensors in different types, and labels representing whether the welding is normal or abnormal; and generating a trained model to estimate an abnormality of welding from a plurality of pieces of detection data obtained by detecting events according to welding with same types of sensors as the plurality of sensors, by taking the plurality of pieces of data for learning as input data, and the labels as training data.

[0013] According to the present invention, it is possible to improve the accuracy of prediction of a welding quality.

[0032] Hereinafter an embodiment of the present invention will be described with reference to the drawings.

(1) System Overview

[0033] FIG. 1 is a diagram illustrating a configuration example of a welding system 100 according to an embodiment. FIG. 2 is a view illustrating a welding example by a welding device 3 of the welding system 100.

[0034] As illustrated in FIG. 1, the welding system 100 includes a welding support device 1, a camera 21, a microphone 22, a welding device 3, a welding power source 4, a storage device 5, and a learning device 6. The camera 21, the microphone 22, the welding device 3, the welding power source 4, and the storage device 5 are connected to the welding support device 1 to enable communication therebetween.

[0035] The welding support device 1 is a computer including a CPU, a GPU, a RAM, a ROM, a non-volatile memory, and an I/O interface. The CPU of the welding support device 1 executes information processing in accordance with a program loaded from the ROM or the non-volatile memory to the RAM.

[0036] The welding support device 1 includes an acquisition unit 11, a conversion unit 12, and an estimation unit 13. These functional units are implemented by the CPU of the welding support device 1 executing information processing in accordance with a program.

[0037] The program may be supplied, for example, via an information storage medium such as an optical disk or a memory card, or may be supplied, for example, via a communication network such as the Internet or a LAN.

[0038] The learning device 6 is also a computer similar to the welding support device 1. The learning device 6 includes an acquisition unit 61 and a learning unit 62.

[0039] The welding support device 1 and the learning device 6 are accessible to the storage device 5. The storage device 5 stores a trained model M generated by the learning device 6, and the trained model M can be read by the welding support device 1.

[0040] As illustrated in FIG. 2, in this embodiment, the welding device 3 is a welding robot that performs arc welding in a groove G formed between two to-be-welded members U, L while advancing a welding torch 31. A molten pool P is formed in a weld zone during arc welding.

[0041] In the example illustrated, the to-be-welded members U, L are arranged in a vertical direction (up-down direction), and the groove G extends in a horizontal direction (front-rear direction). Without being limited to this, the to-be-welded members U, L may be arranged in a horizontal direction.

[0042] The distance between the to-be-welded members U, L is, for example, around 3 to 10 mm. A backing material may be applied or may not be applied to the to-be-welded members U, L. The shape of the groove G is not limited to a V shape as illustrated, and may be an X shape.

[0043] The arc welding is, for example, TIG (Tungsten Inert Gas) welding. Without being limited to this, the arc welding may be MIG (Metal Inert Gas) welding or MAG (Metal Active Gas) welding.

[0044] The welding device 3 performs arc welding while weaving the welding torch 31. For example, when the to-be-welded members U, L are arranged in an up-down direction and the welding direction is the forward direction, to prevent sagging of the molten pool P, weaving of the welding torch 31 is performed in a lower front to upper rear direction.

[0045] The camera 21 captures an image of the weld zone during arc welding to generate image data. Hereinafter the image data generated by the camera 21 is referred to as the “camera image”. The camera 21 is an example of a sensor that detects an event according to arc welding, and the camera image is an example of data for learning and detection data.

[0046] Specifically, the camera 21 captures an image of the molten pool P which is formed in the vicinity of the tip end of the welding torch 31. In addition to the molten pool P, the image capture range of the camera 21 also includes an arc generated from the tip end of the welding torch 31 and a wire (filler metal) extended to the molten pool P.

[0047] The camera 21 is disposed forward or rearward of the welding torch 31, and is moved in a forward direction along with the welding torch 31. To reduce incidence of arc light, the lens of camera 2 is provided with a band-pass filter which allows only near-infrared light in the vicinity of 950 nm to pass therethrough.

[0048] The camera 21 is a video camera that generates video images including a plurality of time series still images (frames). Without being limited to this, the camera 21 may be a still camera that generates a plurality of time series still images by periodical image capturing.

[0049] The microphone 22 records welding sound during arc welding to generate sound data. Hereinafter the welding sound data generated by the microphone 22 is simply referred to as “welding sound”. The microphone 22 is an example of a sensor that detects events according to the arc welding, and the welding sound is an example of data for learning and detection data.

[0050] The welding power source 4 (see FIG. 1) supplies power source for arc welding to the welding device 3. For example, constant current characteristics or constant voltage characteristics are applied to the output characteristics of the welding power source 4. The welding power source 4 includes a voltmeter and an ammeter, and measures the voltage and the current during arc welding.

[0051] Hereinafter the voltage and current data during arc welding measured by the welding power source 4 are simply referred to as the “voltage” and the “current”. The voltmeter and the ammeter included in the welding power source 4 are an example of a sensor that detects an event according to arc welding, and the voltage and the current are an example of data for learning and detection data.

[0052] In this embodiment, the camera 21, the microphone 22 and the welding power source 4 are mentioned as an example of a sensor; however, as long as a sensor can detect an event according to arc welding, the sensor is not limited to these examples.

(2) Learning Phase

[0053] Hereinafter the learning phase implemented in the learning device 6 as a method of generating a trained model according to the embodiment will be described. In this embodiment, so-called multimodal learning, in which a plurality of types of data are used as input data, is performed.

[0054] FIG. 3 is a table for explaining the data set for learning used in the learning phase. The data set for learning includes camera images obtained by capturing a weld zone during arc welding, spectrograms of welding sound and labels representing normal/abnormal. The camera images and the spectrograms are examples of a plurality of pieces of data for learning, and used as input data. The labels are used as training data.

[0055] FIG. 4 is a picture illustrating an example of a camera image. The camera image includes a molten pool, an arc, and a wire. The camera image is generated by the camera 21 capturing an image with a rate of around 100 fps (frame per second), for example. In FIG. 4, for the purpose of description, the left end of the molten pool, the right end of the molten pool, the rear end of the molten pool, the center of the arc, and the tip end of the wire are each denoted by a dotted line or a circle mark. Note that these are explained as feature points in the later-described second embodiment.

[0056] FIG. 5 is a picture illustrating an example of a spectrogram of welding sound. The spectrogram is a graph that is obtained by performing Fourier transform successively with time, and represents welding sound in terms of three dimensions: time, frequency, and strength. In FIG. 5, the horizontal axis represents time, the vertical axis represents frequency, and color (or brightness) represents strength. The welding sound is generated by recording it by the microphone 22 with a sampling rate of around 44.1 kHz, for example.

[0057] As illustrated in FIG. 3, the spectrogram is produced by retrieving welding sound data during a period from a predetermined time before an imaging timing of the camera image to the imaging timing, and is associated with the camera image. For example, when the camera image is captured at a sec., data of welding sound for one second from a−1 to a sec. is retrieved to produce a spectrogram, which is associated with the camera image captured at a sec.

[0058] In this embodiment, as input data (in other words, as an example of data for learning and detection data), camera images and welding sound are used; however, the input data is not limited to these. The input data may include two of more of camera image, welding sound, voltage of welding power source, and current of welding power source, but preferably includes at least one of camera image and welding sound. Note that when the voltage or current of a welding power source is used, a spectrogram is produced in the same manner as for the welding sound.

[0059] Each label represents whether welding is normal or abnormal. For example, when abnormality such as burn through is observed in a camera image obtained by capturing a weld zone during arc welding, the camera image is assigned a label of “abnormality”. In addition, when an abnormality in welding sound in a period is heard, the period may be assigned a label of “abnormality” or a period in which disturbance (an increase in the number of short circuits or a deviation from the average number) of the waveforms of voltage or current occurs may be assigned a label of “abnormality”.

[0060] An abnormality of welding may be determined by an inspection or the like after the arc welding. For example, when a camera image obtained in a period during welding, corresponds to a position of abnormal welding identified from information, such as visual inspection, switching timing for a welding condition, ultrasonic flaw detection and non-destructive inspection using X-rays, the camera image may be assigned a label of “abnormality”.

[0061] Abnormality of welding includes burn through of a molten pool and poor shielding gas performance in addition to an occurrence of a defect (such as a pit, a blowhole, slag inclusion, lack of fusion) in a surface part or inside of a weld zone.

[0062] Note that in this embodiment, the data for learning is detection data such as a camera image and welding sound generated by sensors such as the camera 21 and the microphone 22 included in the welding system 100; however, without being limited to this, the data for learning may be detection data generated by the same types of sensors as the sensors such as the camera 21 and the microphone 22.

[0063] FIG. 6 is a graph illustrating an example of a voltage and a current of the welding power source 4. In FIG. 6, the horizontal axis represents time, and the vertical axis represents the magnitude of voltage and current. The solid line in FIG. 6 represents voltage, and the dashed line represents current. The voltage and the current are obtained by performing analog-to-digital conversion, for example, with a 20 kHz sampling rate (0.05 milliseconds pitch) on the welding power source 4.

[0064] When a wire (filler metal) is used, current and voltage waveforms of droplet transfer called short circuiting transfer are observed. Specifically, when a droplet generated at the wire end is brought into contact with (short-circuited to) a molten pool, a waveform change occurs instantaneously such that the potential difference suddenly drops, and the current suddenly increases.

[0065] FIG. 7 is a graph illustrating an example of the number of short circuits per unit time. In FIG. 7, the horizontal axis represents time, and the vertical axis represents the number of short circuits per unit time (T seconds). Note that the width of the horizontal axis in FIG. 6 corresponds to the unit time (T seconds) approximately.

[0066] Using this result, an interval of increased short-circuit in which the number of short circuits is increased is defined as an abnormal interval. Note that defects of all weld zones may not be detected only by the waveforms of voltage or current in a situation where a defect occurs in some intervals and yet the number of short circuits is not increased, thus an interval of increased short-circuit may be handled as interval of abnormal quality of voltage or current in the later-described second embodiment.

[0067] FIG. 8 is a chart illustrating an example of a trained model M generated in the learning phase. The trained model M is, for example, a convolutional neural network (CNN), and includes periodically arranged convolutional layers, normalization layers, and pooling layers, as well as the output layer provided in the last stage.

[0068] Specifically, the trained model M is a multimodal CNN including a plurality of networks M1, M2. The plurality of networks M1, M2 each include periodically arranged convolutional layers, normalization layers, and pooling layers, and are connected to the output layer provided in the last stage.

[0069] In this embodiment, the trained model M performs convolution on camera images by the first network M1 to extract feature values, and performs convolution on the spectrogram of welding sound by the second network M2 to extract feature values, then combines both feature values immediately before the output layer to estimate the welding abnormality (probability of abnormal welding).

[0070] The output layer is provided with an element corresponding to the welding abnormality. For example, a sigmoid function or a softmax function is used as the element to output a value 0 or greater and 1 or less. The output value of the output layer is used as a welding abnormality, and for example, the output value closer to 0 indicates more normal, and the output value closer to 1 indicates more abnormal.

[0071] FIG. 9 is a flowchart illustrating a procedure example of a learning phase implemented in the learning device 6. The learning device 6 executes the information processing illustrated in FIG. 9 in accordance with a program.

[0072] First, the learning device 6 acquires a data set for learning including camera images, spectrograms of welding sound, and labels representing normal/abnormal (S11: the process as the acquisition unit 61).

[0073] Subsequently, the learning device 6 inputs, as input data, the camera images and the spectrograms of welding sound to a model (untrained model M) to perform calculation by the model, and outputs a welding abnormality from the model (S12 to S14: the processes as the learning unit 62).

[0074] Subsequently, the learning device 6 calculates the error between the welding abnormality as output data outputted from the model, and labels representing normal/abnormal as training data, and performs error back-propagation calculation to reduce the error (S15, S16: the processes as the learning unit 62).

[0075] By repeating the above processes, a trained model M to estimate the welding abnormality from the camera images and the spectrograms of welding sound is generated. The trained model M generated by the learning device 6 is stored in the storage device 5, and used in the below-described inference phase.

(3) Inference Phase

[0076] Hereinafter the inference phase implemented in the welding support device 1 will be described. FIG. 10 is a flowchart illustrating a procedure example of the inference phase. The welding support device 1 executes the information processing illustrated in FIG. 10 in accordance with a program. The inference phase is executed while the welding device 3 performs arc welding.

[0077] First, the welding support device 1 acquires camera images obtained by the camera 21 capturing a weld zone during arc welding, and welding sound during arc welding recorded by the microphone 22 (S21: the process as the acquisition unit 11).

[0078] Subsequently, the welding support device 1 generates a spectrogram of the acquired welding sound (S22: the process as the conversion unit 12).

[0079] Specifically, the welding support device 1 successively acquires, as camera images, a plurality of time series still images (frames) included in the video images generated by the camera 21, and retrieves welding sound during a period from a predetermined time (for example, 1 second) before an imaging timing of each camera image to the imaging timing to convert the welding sound to a spectrogram, and generates a pair of a camera image and a spectrogram of welding sound.

[0080] Subsequently, the welding support device 1 estimates the welding abnormality from the camera images and the spectrograms of welding sound using the trained model M generated in the learning phase and stored in the storage device 5 (S23: the process as the estimation unit 13).

[0081] Specifically, the welding support device 1 inputs, as input data, the camera images and the spectrograms of welding sound to the trained model M to perform calculation by the trained model M, and outputs a welding abnormality from the trained model M.

[0082] Subsequently, when the welding abnormality is greater than a threshold value (S24: YES), the welding support device 1 notifies the welding device 3 of abnormality of welding (S25). Upon receiving notification from the welding support device 1, the welding device 3 performs a predetermined operation such as reduction of speed or stop of forward movement of the welding torch 31.

(4) Effects

[0083] According to the embodiment described above, since the welding abnormality is estimated from the camera images and the spectrograms of welding sound using the trained model M, in-line inspection of quality of welding can be performed with high accuracy.

[0084] In addition, according to the embodiment, when a data set for learning (see FIG. 3) is generated, data for learning such as a camera image only needs to be associated with a label representing normal/abnormal as training data, thus production cost of the training data can be reduced.

[0085] In addition, according to the embodiment, welding sound can be treated as input data to CNN like a camera image by imaging the welding sound to a spectrogram, and a relationship between time, frequency, strength can be learned by performing convolution on the spectrogram.

[0086] In addition, according to the embodiment, the trained model M (see FIG. 8) can complementarily utilize the feature values extracted by the networks M1, M2 via the multimodal CNN. Therefore, abnormality of welding can be detected earlier as compared to when only images are used.

[0087] Note that the plurality of networks M1, M2 may be combined in a layer at a previous stage. Alternatively, a network for spectrogram of voltage or current may be added to implement a multimodal CNN having three or four modes.

[0088] Hereinafter a second embodiment will be described. The same components as in the above embodiment are labeled with the same numeral, and a detailed description may be omitted.

[0089] FIG. 11 is a diagram illustrating a configuration example of a welding support device 1B according to the second embodiment. The welding support device 1B further includes a similarity calculation unit 14 and a reliability determination unit 15 in addition to the acquisition unit 11, the conversion unit 12 and the estimation unit 13.

[0090] In this embodiment, the estimation unit 13 estimates not only a welding abnormality, but also a feature value of detection data. The similarity calculation unit 14 calculates a similarity between the feature value extracted from the data for learning and the feature value estimated by the estimation unit 13. The reliability determination unit 15 determines the reliability of the welding abnormality estimated by the estimation unit 13 based on the calculated similarity.

[0091] FIG. 12 is a table illustrating an example of a data set for learning used in the learning phase. In addition to camera images, spectrograms of welding sound, and labels representing normal/abnormal, the data set for learning further includes the positions of feature points of the camera images, and intervals of abnormal quality of welding sound. The positions of feature points and the intervals of abnormal quality are an example of a feature value, and used as training data.

[0092] As illustrated in FIG. 4, the feature points of a camera image include five points: the left end of a molten pool, the right end of the molten pool, the rear end of the molten pool, the center of an arc, and a wire. Of these, the left end and the right end of the molten pool are represented by x coordinate only, the rear end of the molten pool is represented by y coordinate only, and the center of an arc is represented by x, y coordinates. In short, five feature points are represented by seven variables.

[0093] An interval of abnormal quality of welding sound is, for example, a period in which a disturbance in the waveform of welding sound is observed. Abnormality of quality is assigned, for example, when abnormality of welding cannot be identified and yet the quality of welding sound itself has an abnormality, such as a disturbance in the waveform. Note that an interval of abnormal quality may be partially assigned in the spectrogram.

[0094] FIG. 13 is a chart illustrating an example of a trained model M generated in the learning phase. The trained model M includes a plurality of output layers, to each of which the plurality of networks M1, M2 are connected. Of the plurality of output layers, an output layer 1 corresponds to welding abnormality, an output layer 2 corresponds to position of feature point of camera image, and an output layer 3 corresponds to interval of abnormal quality of welding sound.

[0095] In this embodiment, the trained model M performs convolution on camera images by the first network M1 to extract feature values, and performs convolution on the spectrogram of welding sound by the second network M2 to extract feature values, then combines both feature values immediately before the output layer to estimate the welding abnormality, the positions of feature points of camera images, and the intervals of abnormal quality of welding sound.

[0096] In the learning phase, not only the error between the welding abnormality output from the output layer 1 and the label of normal/abnormal as training data, but also the error between the position of feature point output from the output layer 2 and the position of feature point as training data, and the error between the intervals of abnormal quality output from the output layer 3 and the intervals of abnormal quality as training data are calculated, and error back-propagation calculation is performed to reduce these errors.

[0097] FIG. 14 is a flowchart illustrating a procedure example of the inference phase implemented in the welding support device 1. The same steps as in the above embodiment are labeled with the same numeral, and a detailed description may be omitted.

[0098] First, the welding support device 1 acquires camera images obtained by the camera 21 capturing a weld zone during arc welding, and welding sound during arc welding recorded by the microphone 22 (S21: the process as the acquisition unit 11), and generates a spectrogram of the acquired welding sound (S22: the process as the conversion unit 12).

[0099] Subsequently, the welding support device 1 uses the trained model M to estimate the welding abnormality, the positions of feature points of camera images, and the intervals of abnormal quality of welding sound from the camera images and the spectrograms of welding sound (S33: the process as the estimation unit 13).

[0100] Specifically, the welding support device 1 inputs, as input data, the camera images and the spectrograms of welding sound to the trained model M to perform calculation by the trained model M, and outputs the welding abnormality, the positions of feature points of camera images, and the intervals of abnormal quality of welding sound from the trained model M.

[0101] Subsequently, the welding support device 1 calculates a similarity between the feature value pre-extracted from the data for learning and the feature value estimated in S33 (S34: the process as the similarity calculation unit 14), and determines the reliability of the welding abnormality estimated in S33 based on the calculated similarity (S35: the process as the reliability determination unit 15).

[0102] For calculation of similarity, for example, a method (cosine similarity) of measuring similarity in a vector space is used. For example, the cosine similarity between vector d of estimated data and vector q of data for learning is represented by the Expression (1) below.

[0103] $\begin{array}{cc}\mathrm{Expression}\left(1\right)& \\ \cos \left(\overrightarrow{q},\overrightarrow{d}\right)=\frac{\overrightarrow{q}·\overrightarrow{d}}{❘\overrightarrow{q}❘❘\overrightarrow{d}❘}=\frac{\overrightarrow{q}}{❘\overrightarrow{q}❘}·\frac{\overrightarrow{d}}{❘\overrightarrow{d}❘}=\frac{{\sum }_{i=1}^{❘V❘}{q}_i{d}_i}{\sqrt{{\sum }_{i=1}^{❘V❘}{q}_i^2}·\sqrt{{\sum }_{i=1}^{❘V❘}{d}_i^2}}& \left(1\right)\end{array}$

[0104] Here, the vector d of estimated data represents estimated position coordinates of the feature point of a camera image. Specifically, the vector d of estimated data is represented by (arc center x, arc center y, wire x, wire y, molten pool-left x, molten pool-right x, molten pool-rear end y).

[0105] The vector q of data for learning represents the position coordinates of a feature point randomly selected from the data set for learning. Specifically, the vector q of data for learning is also represented by (arc center x, arc center y, wire x, wire y, molten pool-left x, molten pool-right x, molten pool-rear end y).

[0106] When the value of cosine similarity in Expression (1) above is close to 1, the distributions of estimated data and data for learning are similar to each other, and the reliability of the welding abnormality is high. In contrast, when the value of cosine similarity is smaller than 0.5, for example, the estimated data may not follow the distribution of the data for learning, and the reliability of the welding abnormality is low.

[0107] Subsequently, when the reliability of the welding abnormality is higher than or equal to a threshold value and the welding abnormality is higher than or equal to a threshold value (S35: YES, and S24: YES), the welding support device 1 notifies the welding device 3 of abnormality of welding (S25).

[0108] Specifically, assuming that similarity=reliability, the welding support device 1 determines in S35 whether the similarity is higher than or equal to a threshold value. When the cosine similarity of Expression (1) above is used, the threshold value is 0.5, for example.

[0109] According to the second embodiment described above, a similarity between the feature value extracted from the data for learning and the feature value estimated by the trained model M is calculated, and the reliability of the welding abnormality is determined based on the calculated similarity, thus in-line inspection of quality of welding can be performed with high accuracy.

[0110] Note that in this embodiment, both the positions of feature points of the camera images, and intervals of abnormal quality of welding sound are learned and estimated; however, without being limited to this, either one of the feature values may be learned and estimated. In addition, when the voltage or current of a welding power source is used, intervals of abnormal quality of voltage or current may be learned and estimated.

[0111] Hereinafter a third embodiment will be described. The same components as in the above embodiments are labeled with the same numeral, and a detailed description may be omitted.

[0112] In this embodiment, multimodal learning to estimate welding quality is performed using the time series data for molten pool shape measured from camera images. Thus, even when it is difficult to determine normal/abnormal with camera images, it is possible to determine normal/abnormal with highly accuracy using the time series data for molten pool shape.

(1) Learning Phase

[0113] The learning phase will be described. FIG. 15 is a flowchart illustrating a procedure example of creation of data for learning in the learning phase in this embodiment.

[0114] First, the learning device 6 acquires camera images and current and voltage waveforms which are measured during arc welding for learning, performed by the welding device 3 (S41).

[0115] Subsequently, the learning device 6 estimates the molten pool shape from the camera images using the trained model for feature value extraction (S42). Specifically, the learning device 6 successively estimates the molten pool shape from a plurality of time series camera images to generate time series data for the molten pool shape.

[0116] The trained model for feature value extraction is a trained model to estimate the positions of feature points of the camera images. The trained model for feature value extraction is generated by machine-learning using the camera images as input data, and the positions of feature points as training data. The trained model for feature value extraction uses the same networks to output the positions of feature points, as those included in the trained model explained in the second embodiment (see FIG. 13).

[0117] As illustrated in FIG. 4, the feature points of a camera image include the left end of the molten pool, the right end of the molten pool, the rear end of the molten pool, the center of the arc, and the tip end of the wire. In addition, the feature points of a camera image may include the front end of the molten pool.

[0118] The molten pool shape is, for example, a molten pool width which indicates the width of the molten pool in the right-left direction. The molten pool width is represented by the distance between the position of the left end of the molten pool and the position of the right end of the molten pool which are estimated by the trained model for feature value extraction.

[0119] Alternatively, the molten pool width may be, for example, a molten pool leading distance that indicates the distance by which the front end of the molten pool leads the wire. The molten pool leading distance is represented by the distance between the position of the tip end of the wire and the position of the front end of the molten pool which are estimated by the trained model for feature value extraction.

[0120] Subsequently, the learning device 6 associates the time series data for molten pool shape and the time series data for current and voltage waveforms with abnormal positions (S43). Specifically, the learning device 6 divides the time series data into intervals with a predetermined time (for example, with one second), and assigns each interval a label of normal/abnormal. A data set for learning is produced in the above manner.

[0121] FIG. 16 is a graph illustrating a temporal change example of a molten pool shape. The horizontal axis represents time, and the vertical axis represents the molten pool width. In an experiment, air is blown to a weld zone by a fan to cause poor shielding gas performance, thus a porosity defect is intentionally created. “ON” and “OFF” in FIG. 16 show an ON/OFF state of the fan.

[0122] As illustrated in FIG. 16, a disturbance of the molten pool width is seen in an ON-state period of the fan (the portion surrounded by an ellipse of a two-dot chain line). In other words, occurrence of a disturbance of the molten pool shape due to an abnormality of welding is observed.

[0123] FIG. 17 is a chart illustrating an example of a trained model generated in the learning phase in this embodiment. The trained model includes convolutional layers, RNNs (Recurrent Neural Network), pooling layers, and an output layer.

[0124] Specifically, the trained model is a multimodal RNN including a plurality of networks M3, M4. The plurality of networks M3, M4 each include convolutional layers, RNN, and pooling layers, and are connected to the output layer provided in the last stage.

[0125] The trained model extracts feature values from the time series data for current and voltage waveforms by the first network M3, and extracts feature values from the time series data for molten pool shape by the second network M4, then combines both feature values immediately before the output layer to estimate the welding abnormality.

[0126] In this embodiment, as input data (specifically, as an example of data for learning and detection data), the time series data for current and voltage waveforms and the time series data for molten pool shape are used; however, the input data is not limited to these.

(2) Inference Phase

[0127] The inference phase will be described. FIG. 18 is a flowchart illustrating a procedure example of the inference phase in this embodiment. The welding support device 1 executes the information processing illustrated in FIG. 18 in accordance with a program. The inference phase is executed while the welding device 3 performs arc welding.

[0128] First, the welding support device 1 acquires the camera images obtained by the camera 21 capturing a weld zone during arc welding, and the current and voltage waveforms during arc welding measured by the welding power source 4 (S51: the process as the acquisition unit 11).

[0129] Subsequently, the welding support device 1 estimates the molten pool shape from the camera images using the trained model for feature value extraction, and generates the time series data for molten pool shape (S52). This process is the same process as in S42 of the learning phase.

[0130] Subsequently, the welding support device 1 uses the trained model (see FIG. 17) for abnormality determination generated in the learning phase to estimate the welding abnormality from the time series data for molten pool shape and the time series data for current and voltage waveforms (S53: the process as the estimation unit 13).

[0131] Specifically, the welding support device 1 inputs the time series data for molten pool shape and the time series data for current and voltage waveforms to the trained model to perform calculation by the trained model, and outputs a welding abnormality from the trained model.

[0132] Subsequently, when the welding abnormality is greater than a threshold value (S24: YES), the welding support device 1 notifies the welding device 3 of abnormality of welding (S25). Upon receiving notification from the welding support device 1, the welding device 3 performs a predetermined operation such as reduction of speed or stop of forward movement of the welding torch 31.

[0133] According to the embodiment described above, even when it is difficult to determine normal/abnormal with camera images, in-line inspection of quality of welding can be performed with high accuracy by using the time series data for molten pool shape.

[0134] Although the embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and needless to say, various modifications can be made by those skilled in the art.

[0135] Although various embodiments have been described above with reference to the drawings, needless to say, the present invention is not limited to those examples. It is apparent that various modifications and alterations will occur to those skilled in the art within the scope of the appended claims, and it should be understood that those modifications and alterations naturally fall within the technical scope of the present invention. In a range without departing from the spirit of the invention, the components in the above embodiments may be combined in any manner.

[0136] The present application is based on Japanese Patent Application (No. 2020-174445) filed on Oct. 16, 2020, the entire contents of which are incorporated herein by reference.

• • [0137] 1 welding support device
  • [0138] 11 acquisition unit
  • [0139] 12 conversion unit
  • [0140] 13 estimation unit
  • [0141] 14 similarity calculation unit
  • [0142] 15 reliability determination unit
  • [0143] 21 camera
  • [0144] 22 microphone
  • [0145] 3 welding device
  • [0146] 31 welding torch
  • [0147] 4 welding power source
  • [0148] 5 storage device
  • [0149] 6 learning device
  • [0150] 61 acquisition unit
  • [0151] 62 learning unit
  • [0152] 100 welding system
  • [0153] M trained model
  • [0154] U to-be-welded member
  • [0155] L to-be-welded member
  • [0156] G groove
  • [0157] P molten pool