US 12,697,724 B1Grant
Real time feedback and dynamic adjustment for welding robots
Issue Date:2026-08-04
•20 Claims
•5 Drawing Sheets
Abstract
Systems and methods for real time feedback and for updating welding instructions for a welding robot in real time is described herein. The data of a workspace that includes a part to be welded can be received via at least one sensor. This data can be transformed into a point cloud data representing a three-dimensional surface of the part. A desired state indicative of a desired position of at least a portion of the welding robot with respect to the part can be identified. An estimated state indicative of an estimated position of at least the portion of the welding robot with respect to the part can be compared to the desired state. The welding instructions can be updated based on the comparison.
Metadata
Assignee
- Path Robotics, Inc.
Inventors
- Alexander James Lonsberry
- Andrew Gordon Lonsberry
- Dylan Desantis
- Madhavun Candadai Vasu
- Surag Balajepalli
Application Information
Application Number:US 19/634,812
Filing Date:2026-03-31
Priority Date:2020-07-17
Art Unit:2661
Classifications
IPC:
G06T7/70B23K9/095B23K9/127B23K26/03B25J9/16G01B11/25G05D3/20G06V20/20G06F18/24
Field of Search:
B25J 9/1664B23K 9/0956B23K 9/1274B23K 26/032G01B 11/25G05D 3/20G06T 7/70G06T 2207/10028G06V 20/20G06V 2201/06G06F 18/24
Patent Drawings (5 sheets)
Description
Cross-Reference to Related Applications
[0001] This application is a continuation of U.S. patent application Ser. No. 18/818,355, filed Aug. 28, 2024, which is a continuation of U.S. patent application Ser. No. 18/356,708, filed Jul. 21, 2023, no U.S. Pat. No. 12,109,709, issued Oct. 8, 2024, which is a continuation of U.S. patent application Ser. No. 17/853,045, filed Jun. 29, 2022, now U.S. Pat. No. 11,759,952, issued Sep. 19, 2023, which is a continuation of U.S. patent application Ser. No. 17/379,741, filed Jul. 19, 2021, now U.S. Pat. No. 11,407,110, issued Aug. 9, 2022, which claims the benefit of U.S. Provisional Patent Application Ser. No. 63/053,324, filed Jul. 17, 2020, the disclosures of each of which are hereby incorporated by reference in their entireties.
Technical Field
[0002] This disclosure relates generally to the field of robotic welding. More specifically, this disclosure relates to systems and methods for providing real time feedback and enabling dynamic adjustment for welding robots.
Background
[0003] Welding robots automate the process of welding machine parts. Conventional robotic welding has several challenges. For example, existing methods require a programmer to provide specific instructions (e.g., a welding path) to welding robots to perform robotic welding. Alternatively, handheld devices such as teach pendants can be used to provide specific instructions to welding robots. Known robotic welding systems, however, typically follow welding instructions inflexibly. Therefore, if any changes occur to the welding operation, a portion of the welding robot, the seam, or the part itself and/or if the instructions are not precise, the consequent welding operations can result in bad welds.
[0004] Some existing methodologies use models such as kinematic models or computer aided design (CAD) models to determine instructions for the welding robot. For instance, CAD models can identify an approximate location of a seam to be welded relative to the part. Kinematic models of the robotics system can be used in part to determine the motion plan of the robotic system to move the welding tip in order to deposit material in the desired location. However, these models are not always precise. For example, CAD models may not match the physical dimensions of the part. More specifically, the actual geometric shape or dimensions of the parts may vary slightly from the shape or dimensions of the parts in a CAD model. Such discrepancy can result in bad welds.
[0005] Therefore, there is an unmet need for new and improved systems and methods for providing real time feedback and dynamic adjustment for welding robots to improve the welds in real time and adjust the instructions in real time.
Summary
[0006] In some embodiments, a computer-implemented method for updating welding instructions for a welding robot is described herein. The computer-implemented method can include receiving data of a workspace that includes a part to be welded via at least one sensor. The data can be transformed into a point cloud data representing a three-dimensional surface of the part via a processor. A desired state indicative of a desired position of at least a portion of the welding robot with respect to the part can be identified based on the point cloud data via the processor. An estimated state indicative of an estimated position of at least the portion of the welding robot with respect to the part can be compared to the desired state via the processor. The welding instructions can be updated based on the comparison.
[0007] In some embodiments, the estimated state can be based at least in part on a model of the unwelded part. The model of the unwelded part can include annotations representing the welding instructions for the welding robot. In some embodiments, the estimated state of the part can be based at least in part on the data of the workspace.
[0008] In some embodiments, the at least one sensor can include a first sensor and a second sensor. The computer-implemented method can further comprise synchronizing the first sensor and the second sensor via the processor. In some embodiments, the first sensor can be a laser emitter and the second sensor can be a camera.
[0009] In some embodiments, a plurality of sensors can include the at least one sensor. The computer-implemented method can further compromise fusing respective data received from each sensor of the plurality of sensors to generate fused data, and determine the estimated state based on the fused data.
[0010] In some embodiments, comparing the estimated state to the desired state can comprise implementing a geometric comparator technique. In some embodiments, the computer-implemented method can further comprise updating the point cloud data based at least in part on the desired state. In some embodiments, identifying the desired state can comprise inputting the point cloud data into an artificial neural network to generate the desired state.
[0011] In some embodiments, updating the welding instructions can include adjusting a motion of the welding robot. In some embodiments, updating the welding instruction can include adjusting a motorized fixture associated with the part. In some embodiments, updating the welding instructions can include dynamically updating welding parameters in real time.
[0012] In some embodiments, the computer-implemented method can further comprise detecting a tack weld in a candidate seam on the part to be welded and updating welding parameters based at least in part on the detection of the tack weld. In some embodiments, the computer-implemented method can further comprise detecting that a gap associated with a seam on the part varies and updating welding parameters based at least in part on detection of the variable gap.
[0013] In some embodiments, updating welding instructions can comprise updating welding parameters. The updated welding parameters can include at least one of welder voltage, welder current, duration of an electrical pulse, shape of an electrical pulse, and material feed rate. In some embodiments, receiving the data of the workspace can comprise receiving the data in real time. The computer-implemented method can further comprise monitoring the desired state based at least in part on the data received in real time.
[0014] In some embodiments, identifying the desired state can further comprise implementing a particle filter on the point cloud data. In some embodiments, identifying the desired state can further comprise implementing an online classifier on the point cloud data.
[0015] In some embodiments, the computer-implemented method can further comprise implementing a particle filter and an online classifier on the point cloud data. In some embodiments, the at least one sensor can be a laser emitter. The computer-implemented method can further comprise in response to identifying at least one occluded laser line from the laser emitter, implementing a particle filter on the point cloud data to identify the desired state.
[0016] In some embodiments, a system comprising a sensor, a controller, and a welding robot is described herein. The sensor can be configured to obtain data of a workspace. The workspace can include a part to be welded. A controller can be communicably coupled to the sensor. The controller can be configured to: transform the data to a point cloud data representing a three-dimensional surface of the part, identify a desired state indicative of a desired position of at least a portion of the welding robot with respect to the part based on the point cloud data, compare an estimated state indicative of an estimated position of at least the portion of the welding robot with respect to the part to the desired state, and generate welding instructions based on the comparison. The welding robot can implement the welding instructions to weld the part.
[0017] In some embodiments, an apparatus is described herein. The apparatus can comprise a welding head including at least one laser emitter configured to generate laser lines to sweep an arc of 20 degrees to 60 degrees. The apparatus can also comprise a receiver configured to receive reflected portion of the laser lines. The apparatus can also comprise a controller to generate welding instructions based at least in part on the reflected portion of the laser lines.
Brief Description of the Drawings
[0018] FIG. 1 is a schematic description of a system for precise welding of a part using a welding robot with real time feedback and dynamic adjustment in real time, according to an embodiment.
[0019] FIG. 2 illustrates an example welding head, according to some embodiments.
[0020] FIG. 3 shows an illustration of a welding head attached to a welding robot, according to some embodiments.
[0021] FIG. 4 is a flow diagram of a method for precise welding with real time feedback and dynamic adjustment in real time, according to an embodiment.
Detailed Description
[0022] Non-limiting examples of various aspects and embodiments of the invention are described herein and illustrated in the accompanying drawings.
[0023] Systems and methods for providing real time feedback and dynamic adjustment for welding robots are disclosed herein. Technology described herein can update instructions provided to a welding robot to perform robotic welding in real time based on real time feedback.
[0024] Robots that can be configured to perform operations such as welding, brazing, bonding, a combination thereof, and/or the like are referred to herein as “welding robots.” In contrast to manual operations, welding robots provide significant benefits by reducing production time, cutting cost of labor, improving safety, and conserving materials. “Robotic welding” refers to operations that can be performed by a welding robot, such as welding, brazing, bonding, and/or the like. “Welding robots” as used and described herein can be autonomous or semi-autonomous robots.
[0025] Conventional methods and systems for controlling welding robots have several drawbacks. Typically, a model (e.g., a kinematic model for a welding robot and/or a computer aided design (CAD) model for a part to be welded) can be used to determine an “estimated state” (described in detail below) or sequence of estimated states of the system. For instance, the estimated state could indicate a position of the welding robot (e.g., position of a weld tip of the welding robot) with respect to the part to be welded. Conventional technologies often use the estimated state as a starting point for controlling the welding robot. More specifically, robotic weld programs (e.g., software programs, algorithms, etc.) to control a welding robot are generated based on the estimated state of the system. These robotic weld programs include a weld path (e.g., trajectory for the weld tip) for the welding robot based on the initial state such that the welding robot can perform a welding operation on the part.
[0026] However, there are several challenges associated with relying solely on the model(s). For example, kinematic models may not account for things like backlash, imprecise appendage lengths, or bending of components included in the welding robot. Similarly, CAD models may not include precise dimensions of the part to be welded, for example, when an actual part deviates from a model of that part. Such discrepancies could lead to bad welds. More precisely, the determined estimated state may result in the welding robot being positioned in an inaccurate/imprecise manner relative to the part. That is, the calculated weld path may be inaccurate owing to these discrepancies. Therefore, traditional robotic weld programs can cause poor welds, misplaced welds, collisions amongst components within the system, or other detrimental phenomenon.
[0027] Furthermore, most robotic weld programs are often repeatedly implemented on different units or examples of the same part. Put differently, the same CAD model may be used to determine an estimated state for different units of the same part. Therefore, inconsistencies between different units can lead to bad welds. Similarly, unexpected part placement relative to the welding robot, misaligned parts, poor tolerances, or other similar issues can lead to bad welds.
[0028] Additionally, in some situations, a part may shift, move, or deform during a welding process. More specifically, welding processes often produce high levels of internal stresses to parts which could induce the parts to shift, deform, or otherwise move during a welding process. Such shifts can cause robotic weld programs that are based on estimated state to be imprecise and/or inaccurate thereby causing bad welds. In addition, during processes such as additive manufacturing process or cladding, multiple layers of material may need to be deposited on a part. Accordingly, multiple passes of weld may be needed in order to deposit multiple layers. The estimated state in such a case may therefore change after a single weld pass.
[0029] Thus, real time feedback during welding operations can reduce bad welds. Moreover, using real time feedback to dynamically adjust welding instructions for the welding robots can result in precise welding of parts. However, the accuracy of systems implementing feedback can vary depending on the amount of data that the systems incorporate into feedback.
[0030] Existing feedback system are often primitive providing merely fine adjustments such as fine adjustments to the welding head or adjustments to the welding robot's motion and trajectory so that the welding tip is on the seam to be welded. This does not always ensure that the weld is precise. For instance, the welding instructions are programed in existing feedback systems based typically on two-dimensional drawing or model of the expected seam. Such two-dimensional models may be provided by a user. The welding instructions may include a weld path (e.g., motion and/or trajectory) that the welding robot is to traverse based on the geometry of the seam or gap. Feedback may be provided using a laser-scanner sensor that looks ahead of the welding tip. This can allow existing feedback systems to determine features the welding tip is about to encounter. The laser-scanner looks ahead of the welding tip and provides feedback that generally allows the weld head to follow a seam, as long as the seam geometry remains consistent and visible to the laser-scanner.
[0031] However, there are several drawbacks associated with such existing feedback systems. If the modeled geometry of the seam or the gap is not precise and does not match the actual physical geometry of the seam or the gap, then such feedback systems may not recognize or localize the gaps or seams. Similarly, some portions of the seam or gap may be occluded or may not be completely visible to the laser-scanner during the welding operations. Therefore, existing feedback systems may not recognize such occluded portions during welding operations. Furthermore, since existing feedback systems make minor adjustments to the motion of the welding robot such systems may not have the ability to adjust motion of motorized fixtures (e.g., fixtures that can be actuated to move a part relative to the welding robot).
[0032] Furthermore, in some instances, since the laser-scanner looks ahead to determine features that the welding tip might encounter, the welding robot may be constrained to always point or aim the laser-scanner ahead of the welding operation. These constraints can slow down the system, force a collision when welding a seam, and/or can render a seam to be welded unweldable.
[0033] Therefore, there is a need for sophisticated real time feedback to dynamically adjust welding operations in real time.
[0034] The technology described herein includes sensors to scan a part. In some embodiments, the technology described herein can determine an estimated state of the system, part, welding robot, fixtures, and/or the seam. The estimated state can be estimated position, estimated velocity, and/or estimated acceleration of the system, part, welding robot, fixtures, and/or seam in a universal or relative reference frame. For example, the estimated state can be an estimated position of the welding robot relative to the part. The estimated state can be determined based on sensor input and/or models such as CAD models of the part and/or kinematic models of the welding robot.
[0035] As described above, the sensors can scan the part, fixtures, portions of the welding robot, workspace, etc. The technology described herein can automatically recognize a seam to be welded on the part based on the sensor data. In some embodiments, the technology described herein can localize the seam relative to the part. The technology described herein can then determine a desired state, or sequence thereof, of the system, part, welding robot, fixtures, and/or the seam based on the recognition (and localization) of the seam. The desired state can be desired position, desired velocity, and/or desired acceleration of the system, part, welding robot, fixtures, and/or seam relative to each other. In some embodiments, the technology described herein can compare the desired state to the estimated state and can automatically update a robotic weld program based on the comparison. For instance, the robotic weld program can include a motion/trajectory (e.g., weld path) of the welding robot, welding head, other devices such as motorized fixtures, etc. in order to weld seams on the part precisely as desired without collision prior to the welding operation. An update to the robotic weld program can include dynamic real time adjustment to the motion of the welding robot, welding head, other devices such as motorized fixtures, etc. in order to weld seams on the part precisely as desired without collision during the welding operation.
[0036] As discussed above, the technology described herein can adjust motion/trajectory (e.g., weld path) for a welding robot based on data obtained from sensors. The adjustment can be made dynamically based on the comparison of the expected state with the desired state. Accordingly, in contrast to existing feedback systems, the technology described herein does not require a user to provide the shape of the seams or gaps. Additionally, even if a portion of the seam or the gap is occluded, the technology described herein can still dynamically adjust welding instructions by automatically recognizing seams and/or gaps based on the sensor data.
[0037] In contrast to existing feedback systems, the technology described herein can also operate and/or adjust the motion/trajectory of other devices such as motorized fixtures. For instance, the motorized fixtures can be actuated to move a part to be welded. Since the technology descried herein can also move the motorized fixtures, this can lead to a minimization of energy consumed by the system by limiting total motion within the system. In addition, this can improve or optimize the welding process by optimizing the position and orientation of the seam or gap relative to the welding tip and gravity. Additionally, this can also curtail collisions or possible collisions.
[0038] In contrast to existing feedback systems, the technology described herein can also adjust welding parameters in real time. This in turn can improve the quality of the weld. Some non-limiting examples of welding parameters include welder voltage, welder power or current, material feed rate (if applicable), and other parameters like duration of pulse or shape of electrical pulse, and/or the like. Some non-limiting reasons for adjusting welding parameters alone or in addition to the motion/trajectory of the welding robot and motorized fixture motion include: 1) better welding over or through welds. More specifically, tack welds along a seam usually have material properties that may be different from the material properties of the part(s). Updated welding parameters can account for the shape and geometry of the tack welds; 2) better filling of variable gap sizes by adjusting the feed rate of material and other welding parameters necessary to deposit at that level; 3) insuring good penetration of the weld over different seam geometries; 4) being able to produce the desired weld when for example collision issues interfere with the motion of the welding robot and/or other components of the system; and 5) mitigation, elimination, or alteration of porosity, inclusions, lack of fusion, undercut, overlap, insufficient throat, excessive convexity (i.e. excessive reinforcement, underfill, uneven weld legs, weld splatter, root concavity (i.e. suck-back), hot cracking, or cold cracking.
[0039] Furthermore, since the technology described herein includes a combination of model-based (e.g., determining the estimated state based on a model) and automatic recognition-based (determining a desired state based on the recognition and localization of gaps and seams from sensor data) processing, the technology can operate without a priori information about the shape of the gap or seam.
[0040] FIG. 1 is a schematic description of a system 100 for precise welding of a part 114 using a welding robot 110 with real time feedback and dynamic adjustment in real time, according to an embodiment. A workspace 101 can include a part(s) 114 to be welded. The workspace 101 can be any suitable welding area designed with appropriate safety measures for welding. For example, workspace 101 can be a welding area located in a workshop, job shop, manufacturing plant, fabrication shop, and/or the like. The part(s) 114 can be held, positioned, and/or manipulated in the workspace 101 using fixtures and/or clamps (collectively referred to as “fixtures” 116 herein). The fixtures 116 can be configured to securely hold the part 114. The fixtures 116 can induce forces necessary to limit the movement (e.g., translation and/or rotation) of parts 114 relative to other components of the system 100. For instance, fixtures 116 can be mechanical, electromechanical (e.g., another robotic system), magnetic, or any other means that can fix and/or restrict the movement of the parts 114 relative to other components of the system 100. In some embodiments, the fixtures can be adjustable. In some embodiments, the fixtures 116 can be motorized. Put differently, the fixtures 116 can be actuated to move part(s) 114 during welding process. Therefore, the fixtures 116 can be actuated in such a manner that the position and/or orientation of part(s) 114 relative to other components of the system 100 can be changed. In some embodiments, the fixtures 116 may be located fully within the workspace 101. Alternatively, the fixtures 116 may be located partially within the workspace 101. In another alternative embodiment, the workspace 101 may not include the fixtures 116.
[0041] The welding robot 110 can perform welding operations on the part 114. In some embodiments, the welding robot 110 can be a six-axis robot with a welding arm. The welding robot 110 can be any suitable robotic welding equipment such as Yaskawa® robotic arms, ABB® IRB robots, Kuka® robots, and/or the like. The welding robot 110 can be configured to perform arc welding, resistance welding, spot welding, TIG welding, MAG welding, laser welding, plasma welding, a combination thereof, and/or the like. The welding robot 110 can be responsible for moving, rotating, translating, feeding, and/or positioning welding head, sensor(s) 102, part(s) 114, and/or a combination thereof.
[0042] In some embodiments, a welding head can be mounted on, coupled to, or otherwise attached to the welding robot 110. FIG. 2 illustrates an example welding head 220, according to some embodiments. In some embodiments, the welding head 220 can be a mechanical structure and/or an enclosure that can hold, confine, align, point, or direct at least some of the electrical, optical, or sensor(s) of the welding robot 110. In some embodiments, the welding head 220 can be mounted on, coupled to, or otherwise attached to the welding robot 110 in such a manner that the at least some of the sensor(s) can observe the area or region around the location of the welds during the welding process. In some embodiments, the welding head 220 can include a processor to process sensor information obtained from the sensor(s) included within the welding head 220.
[0043] In some embodiments, the welding head 220 can include laser emitters 202c. In some embodiments, the laser emitter 202c can include polarizing filters. Laser emitters can be laser line(s), laser pattern emitter(s) and/or any other suitable sensing device that is included in, mounted on, or attached to the welding head 220. At least some part of the laser line(s) or pattern(s) can be emitted out of the welding head 220. In some embodiments, laser emitters 202c can be configured to generate laser lines or patterns that sweep and arc of 20 degrees to 60 degrees. Similarly stated, in some embodiments, laser emitters 202c can project line(s) and/or pattern(s) in front of, to the sides of, and/or at least partially behind the welding head 220. In some embodiments, the laser emitters 202c can be motorized to generate laser lines or patterns that sweep across an area of the part 112. This could enable the laser emitters 202c to scan a large area of the part. For instance, the laser emitter 202c itself could be configured to rotate to sweep the area. Additionally or alternatively, a mirror positioned in front of the lasers can be motorized to perform the sweeping motion. For example, the mirrors can have one or two degrees of freedom.
[0044] In some embodiments, the welding head 220 can optionally include laser controller(s) that can be operable to control power delivery to the laser emitters 202c. The laser controller(s) can be configured to turn the lasers on and off. The laser controller(s) can be configured to transmit feedback associated with laser emitters to one or more electronic devices coupled to laser emitter(s) 202c. In some embodiments, a receiver can be configured to receive reflected portions of the laser lines or patterns emitted from the laser emitters 202c. In some embodiments, the laser controller(s) can be configured to additionally function as the receiver.
[0045] One or more optical sensor(s) 202d can be included in the welding head 220. Some non-limiting examples of optical sensor(s) 202d include cameras, lenses, lens filters, any suitable optical device, and/or any suitable opto-electronic device. In some embodiments, filters can be used for filtering out specific wavelengths of light observable to the optical sensor(s) 202d. In some embodiments, filters can protect the optical sensor(s) 202d from different light emission during the welding process. In some embodiments, filters can enhance the laser emission from the laser emitter(s) 202c, for example light polarizing filters can be used. In some embodiments, the welding head 220 can include at least two optical sensor(s) 202d mounted on, attached to, or otherwise coupled to the welding head 220. The optical sensor(s) 202d can observe the welding head 220, the welding process, and the laser light emitted from laser emitter(s) 202c.
[0046] In some embodiments, the welding head 220 can include thermal cameras and/or thermal sensors that sense temperature. In some embodiments, ambient lights and associated controllers can be used to connect to light sources that may be disposed and/or otherwise positioned within the welding head 220 to illuminate the workspace 101.
[0047] The welding head 220 can include air blades 224 and can be used to protect portions of the welding head 220 from particulates and/or emissions from the welding process. For example, an exit air port can be positioned near the optical sensor(s) 202d and/or the laser emitter(s) 202c so as to protect the optical components. Additionally or alternatively, one or more magnets can be positioned within the welding head 220 to pull particulates and/or emissions from the welding process away from the optical components of the optical sensor(s) 202d and/or the laser emitter(s) 202c.
[0048] In some embodiments, the welding head 220 can include one or more fiducials, such as reflective markers, LED light emitters, stickers with identifiable patterns, etc. Such fiducials can be used to track the position and/or the orientation of the welding head 220 by one or more global sensor(s) (e.g., sensor 102a and 102b in FIG. 1 ). By tracking the position and/or the orientation of the welding head 220, a desired position and/or orientation of the welding head 220 can be determined.
[0049] In some embodiments, the welding head 220 can include force feedback sensors to detect force and/or torque applied to the welding head 220. In some embodiments, ultrasonic sensors can be coupled to, mounted on, or otherwise attached to the welding head 220. Ultrasonic sensors can help avoid collision. Additionally, ultrasonic sensors can sense surfaces of objects around the welding head 220 and can therefore be used to determine a desired state of the welding head 220. In some embodiments, inertial measurement unit(s) (e.g., accelerometers and/or gyroscope(s)) can provide an estimate of acceleration and/or orientation of at least a portion of the welding robot 110. The welding head 220 can include a welding torch 222 to perform the welding operation.
[0050] FIG. 3 shows an illustration of a welding head 320 (e.g., structurally and/or functionally similar to welding head 220 in FIG. 2 ) attached to a welding robot 310 (e.g., structurally and/or functionally similar to welding robot 110 in FIG. 1 ) by means of clamping to a welding torch 322 at the end of the welding robot 310.
[0051] Referring back to FIG. 1 , sensors 102a, 102b, 102c, 102d, . . . (collectively referred to as sensor(s) 102) can capture data associated with the workspace 101. In some embodiments, one or more sensor(s) 102 can be a mounted to the robot 110 or otherwise can be integral to the workspace 101. For example, the one or more sensor(s) (e.g., sensors 102c and 102d in FIG. 1 ) can be attached to or otherwise can be a part of the of the welding robot 110. More specifically, the sensor(s) (e.g., sensors 102c and 102d in FIG. 1 ) can be a part of, attached to, or mounted on a welding head (as described in FIG. 2 ) of the welding robot 110. In some embodiments, sensors 102c and 102d can be laser emitters, optical sensors (e.g., cameras), or a combination thereof.
[0052] In some embodiments, one or more global sensor(s) (e.g., sensor 102a and 102b in FIG. 1 ) can be mounted on another robot (not shown in FIG. 1 ) positioned within the workspace 101. For example, a robot may be operable to move (e.g., rotational and/or translational motion) such that the sensor(s) 102a and 102b can capture image data of the workspace 101 from various angles. In some embodiments, one or more sensor(s) 102a and 102b can be portable device such as a handheld computer tablet, a smartphone with camera, or a digital camera. Additionally or alternatively, one or more sensor(s) 102a and 102b can be laser emitters.
[0053] In some embodiments, the sensor(s) 102 described herein can be configured to observe and collect data of all regions including the region around the welding tip. For example, multiple laser lines (e.g., from multiple laser emitters) and/or optical sensors can effectively surround the welding tip, such as forming a closed polygon.
[0054] Sensor(s) 102 can determine information associated with the desired state of the part 114, fixtures 116, welding robot 110, and/or the seam. As discussed above, desired state can include the desired position and/or desired orientation of the part 114, fixtures 116, welding robot 110, and/or the seam in the physical space relative to each other. Additionally or alternatively, desired state can include desired velocity, acceleration, forces, and/or torques associated with the part 114, fixtures 116, welding robot 110, and/or the seam. Sensor data can include multiple sets of state such that each state in the set of states can correspond to a state at a different point in time.
[0055] The sensor(s) 102 can be operable to capture 2D and/or 3D images/data of the part 114, fixture 116, portion of the robot 110, and/or workspace 101. In some instances, the sensors 102 can be operable to image and/or monitor a weld laid by the robot 110, before, during, and/or after weld deposition. In some embodiments, the sensor(s) 102 can generate a point cloud representation of the workspace 101.
[0056] FIG. 1 illustrates two sensors 102c and 102d within the robot 110 and two global sensors 102a and 102b solely for illustrative purposes. It should be readily understood that any suitable number of sensors can be included in the robot 110. Similarly, any suitable number of sensors can be positioned in the workspace 101 as global sensors.
[0057] The sensor(s) 102 can be communicatively coupled to the controller 108. The sensor(s) 102 and/or the controller 108 can be operable to process data from the sensor(s) 102 to assemble two-dimensional data and three-dimensional data from multiple sensors 102 to generate point cloud data. The controller 108 can include one or more processors (e.g., CPU). The processor(s) can be any suitable processing device configured to run and/or execute a set of instructions or code, and can include one or more data processors, image processors, graphics processing units, digital signal processors, and/or central processing units. The processor(s) can be, for example, a general purpose processor, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), and/or the like. The processor(s) can be configured to run and/or execute application processes and/or other modules, processes and/or functions associated with the system 100.
[0058] In some embodiments, the controller 108 can synchronize the sensor(s) 102. For instance, the controller 108 can synchronize optical sensor(s) and laser emitter(s) such that when the sensor(s) capture images of the part(s) 112 and/or workspace 101, each captured image includes only a single laser pattern such as a line. Put differently, multiple laser lines originating from different laser emitters can be challenging for optical sensors to distinguish. Accordingly, the controller 108 can be configured to strobe or alternate laser lines such that only a single line is present at any given time. Similarly, the controller 108 can synchronize optical sensor(s), welding head, and laser emitter(s) such that the laser emitter(s) are turned on and an image of the laser response captured when the welding head is not operating in pulse mode (e.g., mode in which an electric arc or electric current that induces the substrate, filler material, or both to melt occurs in pulses through time). Such synchronization can produce images with less light pollution induced by the welding process, for example, by capturing images in between periods in which the welder is active (e.g., between arc pulses).
[0059] In some embodiments, the controller 108 can process the data obtained from the sensor(s) 102. For example, the controller can identify and generate laser lines and/or patterns emitted by laser emitter(s) such as disclosed in U.S. Pat. No. 10,551,179, the entire disclosure of which is hereby incorporated by reference. The laser lines and/or patterns as captured by optical sensors can vary in the presence other components of the system 100, shadows, different light sources, light emitted from welding processes, etc.
[0060] In some embodiments, the controller can generate three-dimensional point cloud data. Point cloud data can represent three-dimensional surface of parts 112, fixtures 116, and other components of system 100 that are visible to the welding head of the welding robot 110. Point cloud data can be generated, for example, using laser and/or light triangulation techniques, as Structure-From-Motion, and/or Stereo Based approaches. The three-dimensional data may be collected using laser finding techniques.
[0061] In some embodiments, one or more images (e.g., image data captured by the sensor(s) 102 at a particular orientation relative to part 114) may be overlapped together to reconstruct and generate three-dimensional image data of the workspace 101. The three-dimensional image data can be collated in a manner such that the point cloud generated from the data can have six degrees of freedom. For instance, each point on the point cloud may represent an infinitesimally small position in three-dimensional space. The sensor(s) 102 can capture multiple images of the point from various angles. These multiple images can be collated to determine an average image pixel for that point. The averaged image pixel can be attached to the point. For example, if the sensor(s) 102 are color cameras having red, green, and blue channels, then the six degrees of freedom can be {x-position, y-position, z-position, red-intensity, green-intensity, and blue-intensity}. If the sensor(s) 102 are black and white cameras with black and white channels, then four degrees of freedom may be generated.
[0062] The controller 108 can be configured to perform seam recognition based on the point cloud data. In some embodiments, the controller 108 can localize, classify, and/or measure seams, separating volumes, and/or welds. In some embodiments, the controller 108 can implement a neural network to recognize and/or classify a seam to be welded. For instance, the neural network can be configured to implement pixel-wise and/or point-wise classification on the point cloud data in order to recognize and classify a seam to be welded. In this manner, the controller 108 can generate recognized seam data that encodes the position of seams, separating volumes, and/or welds. The controller 108 can determine a desired state of the part 112, fixture 116, welding robot 110, and/or seam based on the seam recognition and classification.
[0063] In some embodiments, the controller can determine an estimated state of the part 112, fixture 116, welding robot 110, and/or seam based on one or more a priori models (e.g., kinematic models of welding robot 110, CAD models of part 112, etc.). For example, CAD models of the part 112 can include annotations representing one or more seams on the part to be welded. Based on the geometry and/or shape of the part 112 and seams, the controller 108 can determine an estimated state. The controller 108 can then compare the estimated state to the desired state. For instance, the controller 108 can implement geometric comparator techniques to compare the estimated state to the desired state. The controller 108 can then correct for inaccuracies in data from sensor(s) 102 or state estimations (e.g., desired state and estimated state estimations), to correct for bending, movement, or deformation of part(s) 112 relative to the system 100 that may be induced by the welding process. In this manner, the controller 108 can provide dynamic adjustments to welding instructions.
[0064] Geometric comparator techniques takes into account geometric data (e.g., expected, modeled, and/or predicted data) of the from part(s) 112, seam(s), etc., measured state information of the part(s) 112, robot 110, fixture(s) 116, etc., and also three-dimensional point cloud data and uses this data. Geometric comparator techniques can use this data to generate a more accurate estimate of state(s) (e.g., desired state) of the part(s) 112, seam(s) 116, robot 110, weld(s), etc. Some non-limiting examples of geometric comparator techniques generally can include determining an estimated state from the models, determining a desired state from the three-dimensional point cloud, and measuring and quantizing the difference between the desired state and the estimated state.
[0065] In some embodiments, an accurate estimate of the desired state can be determined by implementing, Kalman filters, particle filters, a combination thereof, and/or the like. For instance, particle filters at some given time-step, can include several particles representative of possible or feasible states. The possible or feasible states can be associated with simulated point cloud data, stimulated image data, or combination thereof for each of the particles. The estimated state can then be determined by comparing the stimulated point cloud data and/or the stimulated image data (collectively referred as “simulated data”) with the actual point cloud data and/or actual image data (collectively referred as “actual data”). This helps determine the most probable particle (state) that the system is currently in.
[0066] In some embodiments, the controller 108 can implement a neural network such as Convolutional Neural Network (CNN), Artificial neural network (ANN) to estimate a desired state. In some embodiments, the neural network can take image data, point cloud data, or a combination thereof, and output an encoding of the input. For instance, the encoding can be into a smaller dimensional latent space. In some embodiments, the output from a particle of the particle filter (e.g., most probable particle that the system is currently in) can be provided as an input to the neural network. For example, a stimulated point cloud data and/or image data may be provided as an input to the neural network. The output of this input (i.e., the encoding of this input) can be compared to the output from the neural network of actual point cloud data and/or actual image data. For instance, the output of the stimulated data can be compared with the output of the actual data by performing a dot product of the two. Additionally or alternatively, the neural network can input output from one of the particle of the particle filter as well as the actual data at the same time. The neural network can then generate a score that represents the similarity between the two data. Additionally or alternatively, any suitable method may be used to compare the simulated data to the actual data.
[0067] In some embodiments, different CNNs can be used to process image data from optical cameras, thermal cameras, and/or other sensors. Alternatively, the controller can be configured to fuse image data from various sources. For example, the controller 108 can fuse data from various sensor(s) 102 such as optical sensor(s), laser emitter(s), and other sensor(s). By fusing the data from various sources, the controller 108 can provide accurate feedback to the welding robot 110 in order to perform precise welding. For instance, by receiving data of the welding operation from two or more sensors can provide a wide-angle view of the part and the weld tip during the welding process. In contrast, when a single sensor such as a laser emitter is used to provide data, it is possible that some portion of the seam that is being welded may be occluded from the laser emitter. Such occlusion can lead to large deviations from the desired state. With complete view of the part during the welding process, the controller can accurately identify the desired state and make updates to it in real time. In some embodiments, in order to fuse data from various sources, in the CNN where branches of networks converge are combined. The CNN(s) can be trained, for example, using cost or error functions, parameter optimization or parameter adjustment, gradient descent methods, and/or any other suitable technique.
[0068] In some embodiments, the controller 108 can implement a Neural Network in order to analyze the weld state (e.g., state of the part(s) 112, fixture(s) 116, welding robot 110, and/or seam during the welding process). For instance, as discussed above a neural network can be implemented to compare simulated data with the actual data. Additionally or alternatively, a neural network can be implemented to classify seams or gaps to be welded. In some embodiments, a neural network can be implemented to classify tack welds or other welds that exist on the seam. The neural network can be structurally and/or functionally similar to a Artificial Neural Network (ANN), Recurrent Neural Network (RNN) and/or Convolutional Neural Network (CNN) or any other suitable neural network. For example, ANN elements that perform different classification, regression, or generative tasks, may feed into other ANNs or other intermediate processing methods or filters. The ANN can be configured to derive, generate, and/or originate weld state information based on data originally produced by the welding head, sensor(s) 102, or combination thereof. The ANN can be trained, for example, using cost or error functions, parameter optimization or parameter adjustment, gradient descent methods, and/or any other suitable techniques.
[0069] In some embodiments, the welding robot 110, part(s) 112, fixtures 116, and/or other components of the system 100 can be controlled by a robotic welding program executed on the controller 108. The robotic weld program can be configured to transmit welding instructions to the welding robot 110, part(s) 112, fixtures 116, and/or other components of the system 100 so as to weld seams or separating volumes. In some embodiments, the robotic weld program can be generated prior to the commencement of the welding process or execution of the robotic weld program itself. The robotic weld program can be dynamically altered, interrupted, deleted, modified, or added to by the controller 108 based on comparison of the desired state and estimated state. For instance, the robotic weld program can be updated, created, altered, deleted, amended, or otherwise changed specific to the motion/trajectory of the welding robot 110 and/or other motorized components (e.g., motorized fixture(s) 116) within the system 100. Additionally or alternatively, the robotic weld program can be updated, created, altered, deleted, amended, or otherwise changed specific to the welding process (e.g., angle of the weld tip relative to the part, velocity of weld tip, acceleration of weld tip, etc.). As discussed above, the controller 108 can update, create, alter, delete, amend, or otherwise change the robotic weld program based on the comparison of the estimated state to the desired state. For instance, the weld tip may be moved from an estimated position to a more accurate desired position.
[0070] Some non-limiting techniques implemented by the controller 108 to update the robotic weld program include a) implementing a linear time invariant controller employing control theory methods, tuning, and modeling, that uses error based on modeled and observed state information for the robot 110, fixture(s) 116, part(s) 114, seam, weld, etc.; b) implementing a non-linear classical controller that can be build, for example, on either Gain Scheduling, Feedback Linearization, or Lyapunov, and/or any other suitable methods that uses error based on modeled and observed state information for the robot 110, fixture(s) 116, part(s) 114, seam, weld, etc.; c) implementing optimal motion controller(s), which can include any applicable optimal controllers, that uses error based on modeled and observed state information for the robot 110, fixture(s) 116, part(s) 114, seam, weld, etc.; d) implementing artificial neural network controllers which can be built and trained neural networks that are constituted based on mathematical elements most typically associated with ANNs and can use error based on modeled and observed state information for the robot 110, fixture(s) 116, part(s) 114, seam, weld, etc.
[0071] In some embodiments, the controller 108 can be configured to recognize and measure root gap, hole, void, and weld and update robotic weld program based on recognition in order to produce more precise welds. The two-dimensional images and the three-dimensional point cloud data from the sensor(s) 102 can be used to recognize the root gap, hole, void, or weld and further can be used to adapt and/or update the robotic weld program to account for the root gap, hole, void, or weld. In some embodiments, in a first mode, the controller 108 can first recognize the root gap, hole, void, or weld and can then subsequently update the robotic weld program. Alternatively, in a second mode, the controller 108 can recognize the root gap, hole, void, or weld and can update the robotic weld program simultaneously.
[0072] In the first mode, the controller 108 can break down the task of recognition, classification, and measurement of root gaps, holes, voids, welds, or any obstruction along the seams separately. This can provide an indication on how to adapt and/or update the robotic weld program. In this mode, a user can see and understand the decisions that the controller 108 makes.
[0073] Recognition and measurements of root gaps, holes, voids along the seam can be accomplished in part or in whole via trained neural networks. The input data to these neural networks can be point cloud data and/or two-dimensional image data. The data flows into a one or more networks which then classifies regions or boundaries of holes, voids, or root gap. In some embodiments, the neural network can determine the size of the holes, voids, or root gap. The input to this neural network may include a model of the part(s) 112 or regions on the part(s) 112. For example, the input to the neural network can be a CAD model of a part(s) 112 that has had its surface sampled and turned into point cloud. In some embodiments, these neural networks can be any suitable neural network such as Dynamic Graph CNN, point-net, etc. The input data can be presented to the neural network in any suitable form (e.g., as unordered list of vectors). The input data can be pre-processed (e.g., using normalization).
[0074] In some embodiments, the input data can be provided to the neural network at once. Alternatively, the input data can be provided to the neural network incrementally. The neural networks can be trained in any suitable manner. In some variations, the neural networks can include long short-term memory (LSTM) modules, or other similar modules meant to deal with time-varying or sequential data. The output could be in any suitable form. For example, the output could include classification of gap associated for each original three-dimensional points and/or also a subset of the three-dimensional points and/or can be associated with pixels of the two-dimensional images. The neural networks can be trained to output other information simultaneously, for example surface roughness measurements, and are not necessarily limited to just handle the recognition and measurement tasks alone.
[0075] Adaptation and/or updating the robotic weld program can also occur in part via trained neural network(s). This neural network, or set of neural networks, can take in the resultant classification output described above, information about what the desired weld is, and where it should be. This neural network(s) can then output at least information about orientation and/or location of the weld tip relative to the seam. The network may also be trained such that weld parameters, for example voltage, current, arc-length, wire/wire material, substrate material and thickness, and orientation of the seam relative to gravity can be either inputs, outputs, or some combination of both. For example, a user may want to constrain the voltage, current, and arc-length, (inputs the network) but not the orientation of the seam relative to gravity (output). Alterations or adaption of weld parameters like current, voltage, or arc-length can be useful for changing the shape of the welds or handling transition between substrates like welds that have been previously deposited. The neural network may be trained to take in a range, envelope, or bounds of parameters. For example, there could be two inputs for voltage that bound the allowable range. The network may be trained to output over a range, envelope, or bounds of parameters. For example, the network could have n number of outputs discretized over an associated range of some parameter or parameters. The network may be trained to indicate in some manner if the desired weld output is feasible or not. In some embodiments, the network may also output alternative similar welds that are feasible, for example instead of fillet weld of some radius it could output a larger weld in order to cover some larger gap.
[0076] The neural network can also be trained to also output an indication of the resultant weld quality, weld shape, weld penetration, measure of porosity, or a combination thereof. In some embodiments, the neural network can be any suitable neural network such as Dynamic Graph CNN, point-net, etc. The input data can be presented to the neural network in any suitable form (e.g., unordered list of vectors). The input data can be pre-processed (e.g., using normalization).
[0077] In some embodiments, the input data can be provided to the neural network at once. Alternatively, the input data can be provided to the neural network incrementally. The neural networks can be trained in any suitable manner. In some variations, the neural networks can include LSTM modules or other similar modules meant to deal with time-varying or sequential data. The output could be in any suitable form. The neural network can be trained to output other information simultaneously, for example, the probability of a successful weld.
[0078] In the second mode, the recognition and adaption can occur in one shot. In contrast to the first mode, there is no intermediary recognition or classification step in the second mode. The inputs to the neural network in this mode can include a model of the part(s) 112 or regions on the part(s) 112. For example, the input to the neural network can be a CAD model of a part(s) 112 that has had its surface sampled and represented as point cloud. The input to the neural network can further include information about what the desired weld is and where it should be. The output includes at least information about orientation and/or location of the weld tip relative to the part.
[0079] In some embodiments, the neural network can also be trained such that weld parameters, for example voltage, current, arc-length, wire/wire material, substrate material and thickness, and orientation of the seam relative to gravity can be either inputs, outputs, or some combination of both. For example, the user may want to constrain the voltage, current, and arc-length, (inputs the network) but not the orientation of the seam relative to gravity (output). Alterations or adaption of weld parameters like current, voltage, or arc-length can be useful for changing the shape of the welds or handling transition between substrates like welds that have been previously deposited. The neural network may be trained to take in a range, envelope, or bounds of parameters. For example, there could be two inputs for voltage that bound the allowable range. The network may be trained to output over a range, envelope, or bounds of parameters. For example, the network could have n number of outputs discretized over an associated range of some parameter or parameters. The network may be trained to indicate in some manner if the desired weld output is feasible or not; it may also output alternative similar welds that are feasible, for example instead of fillet weld of some radius it could output a larger weld in order to cover some larger gap. The network may be trained to also output an indication of the resultant weld quality, weld shape, weld penetration, measure of porosity, or a combination thereof. In some embodiments, the neural network can be any suitable neural network such as Dynamic Graph CNN, point-net, etc. The input data can be presented to the neural network in any suitable form (e.g., unordered list of vectors). The input data can be pre-processed (e.g., using normalization).
[0080] In some embodiments, the input data can be provided to the neural network at once. Alternatively, the input data can be provided to the neural network incrementally. The neural networks can be trained in any suitable manner. In some variations, the neural networks can include LSTM modules or other similar modules meant to deal with time-varying or sequential data. The output could be in any suitable form. The neural network can be trained to output other information simultaneously, for example, the probability of a successful weld.
[0081] In some embodiments, Simultaneous Localization and Mapping can be used to fill in holes, gaps, or surface regions in the parts or seams (e.g., if these were not initially scanned completely). In some embodiments, Simultaneous Localization and Mapping can be used to continue to localize in the case that CAD models are incomplete, and/or if a scan of a part or seam misses a portion of the part and/or includes ambiguous or unreliable data.
[0082] In some embodiments, the controller 108 can continuously update models (e.g., kinematic models and/or CAD models) for example, by adding point cloud data to these models. In some embodiments, the point cloud data can include information associated with the addition of a weld that has been deposited or created. Such data can, for example, be collected via the laser emitter(s) illuminating a portion of the part behind the weld tip of the robot 110 as it travels. “Behind” here can refer to the direction substantially opposite to the travel direction of the weld tip and/or a portion of the part recently passed by the weld tip.
[0083] The controller 108 can also add weld state data to models, for example, adding information about weld state based on point(s) or region(s) of the surface as observed using three-dimensional point cloud data. In this way, weld state data can be assigned such that information includes not only the three-dimensional geometry (from the three-dimensional point cloud), but also to some extent information about the weld (e.g., based on CNN/ANN derived weld models). In some embodiments, the CAD model can be continuously updated and can include notions of time or some notion of sequence. Additionally or alternatively, CAD model can be updated based on scanned data of the part(s) 112 during, between, or after the welding process.
[0084] Although system 100 can operate with little or no user interaction, in some embodiments, system 100 can optionally include a user interface 106 that can allow users to provide weld parameters and/or update weld parameters. The user interface 106 can enable a user to interact with the system 100. Some non-limiting examples of the user interface 106 include menu-driven interface, graphical user interface (GUI), touchscreen GUI, a combination thereof, and/or the like.
[0085] FIG. 4 is a flow diagram of a method 400 for precise welding with real time feedback and dynamic adjustment in real time, according to an embodiment. At 402, the method includes receiving data of a workspace (e.g., workspace 101 in FIG. 1 ) including a part (e.g., part(s) 112 in FIG. 1 ) to be welded. The data can be sensor data from sensors (e.g., sensor(s) 102 in FIG. 1 ) such as laser emitter(s), optical sensor(s), and/or a combination thereof. In some embodiments, these sensor(s) can be synchronized before initializing the method 400. In some embodiments, data from different sensor(s) can be fused together and then analyzed.
[0086] At 404, the method includes transforming the data from the sensor into point cloud data. For example, the data from the sensors can be two-dimensional data. This data can be transformed into a three-dimensional point cloud data representing the three-dimensional surface of the part.
[0087] At 406, the method includes determining desired state based on the point cloud data. The desired state can be indicative of a desired position of at least a portion of the welding robot (e.g., welding robot 110 in FIG. 1 ) relative to the part. For instance, the desired state can be indicative of the desired portion of a weld tip included in the welding robot relative to the part. In some embodiments, an artificial neural network can be implemented on the point cloud data to determine the desired state. In some embodiments, the desired state can be determined by recognizing a seam to be welded on the part. In some embodiments, the desired state can be determined by localizing the seam relative to the part.
[0088] In some embodiments, the desired state can be determined by implementing an online classifier on the point cloud data. However, it may be challenging to determine the desired state of portions of a part that may be occluded from the view of a laser emitter(s) using the online classifier. In some embodiments, in response to determining such occlusions, the desired state can be determined by implementing a particle filter. In some embodiments, the desired state can be determined by implementing a combination of online classifier and a particle filter.
[0089] At 408, the method includes comparing an estimated state to the desired state. The estimated state may be indicative of an estimated position of at least a portion of the welding robot (e.g., weld tip) relative to the part. The estimated state may be determined based on models such as kinematic models and/or CAD models. A geometric comparator technique may be implemented to compare the estimated state to the desired state.
[0090] At 410, the method includes updating welding instructions based on the comparison. The welding instructions can be included in a robotic weld program and updating the welding instructions can include updating the robotic weld program. In some embodiments, updating the welding instructions can include updating a motion of the robot (e.g., moving the robot from a previous position to a new position to reach the desired state). In some embodiments, updating the welding instructions can include updating a motion of motorized fixtures (e.g., fixture(s) 116 in FIG. 1 ). In some embodiments, updating the welding instruction can including updating weld parameters in real time. The welding instructions can also be updated based on a detection of a tack weld and/or detection of a variable gap on the seam to be welded.
[0091] It should be readily understood that method 400 can be continuously repeated in any suitable manner. For example, prior to welding operations, welding instructions included in the robotic weld program can include a motion/trajectory (e.g., weld path) for the welding robot and/or the motorized fixtures based on an initial estimated state or an initial desired state. During the welding process, the sensors can provide data of the workspace (e.g., part being welded, seam being welded, weld tip, etc.). Based on this data, an updated desired state can be determined. This process can be continued until the welding process comes to an end. Put differently, the weld path can be continuously monitored and updated through the welding process.
[0092] While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. For example, some embodiments described herein reference a single light source, a single detector, and/or a single pattern. It should be understood, however, that multiple light sources can illuminate a part. Each light source can illuminate the part with a similar pattern, or different patterns. Patterns can be detected by multiple detectors. In embodiments with multiple light sources, it may be advantageous for each light source to illuminate the part with a different pattern such that detector(s) and/or compute devices processing images captured by detectors can identify which light source projected which pattern.
[0093] Furthermore, although various embodiments have been described as having particular features and/or combinations of components, other embodiments are possible having a combination of any features and/or components from any of embodiments where appropriate as well as additional features and/or components. For example, although not described in detail above, in some embodiments, methods of determining a shape of a portion of a part may include a calibration phase during which distortion of the detector(s), the lens(es) on said detector(s), the distortion in the combination of detector(s) and lens(es), and/or the relative position of the camera(s) to a test surface or fixture onto which a pattern(s) is projected are determined.
[0094] Some embodiments described herein relate to methods and/or processing events. It should be understood that such methods and/or processing events can be computer-implemented. That is, where method or other events are described herein, it should be understood that they may be performed by a compute device having a processor and a memory. Methods described herein can be performed locally, for example, at a compute device physically co-located with a robot or local computer/controller associated with the robot.
[0095] Memory of a compute device is also referred to as a non-transitory computer-readable medium, which can include instructions or computer code for performing various computer-implemented operations. The computer-readable medium (or processor-readable medium) is non-transitory in the sense that it does not include transitory propagating signals per se (e.g., a propagating electromagnetic wave carrying information on a transmission medium such as space or a cable). The media and computer code (also can be referred to as code) may be those designed and constructed for the specific purpose or purposes. Examples of non-transitory computer-readable media include, but are not limited to: magnetic storage media such as hard disks, floppy disks, and magnetic tape; optical storage media such as Compact Disc/Digital Video Discs (CD/DVDs), Compact Disc-Read Only Memories (CD-ROMs), and holographic devices; magneto-optical storage media such as optical disks; carrier wave signal processing modules, Read-Only Memory (ROM), Random-Access Memory (RAM) and/or the like. One or more processors can be communicatively coupled to the memory and operable to execute the code stored on the non-transitory processor-readable medium. Examples of processors include general purpose processors (e.g., CPUs), Graphical Processing Units, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Digital Signal Processor (DSPs), Programmable Logic Devices (PLDs), and the like. Examples of computer code include, but are not limited to, micro-code or micro-instructions, machine instructions, such as produced by a compiler, code used to produce a web service, and files containing higher-level instructions that are executed by a computer using an interpreter. For example, embodiments may be implemented using imperative programming languages (e.g., C, Fortran, etc.), functional programming languages (Haskell, Erlang, etc.), logical programming languages (e.g., Prolog), object-oriented programming languages (e.g., Java, C++, etc.) or other suitable programming languages and/or development tools. Additional examples of computer code include, but are not limited to, control signals, encrypted code, and compressed code.
[0096] Where methods described above indicate certain events occurring in certain order, the ordering of certain events may be modified. Additionally, certain of the events may be performed concurrently in a parallel process when possible, as well as performed sequentially as described above. Although various embodiments have been described as having particular features and/or combinations of components, other embodiments are possible having a combination of any features and/or components from any of the embodiments where appropriate.
[0097] The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that specific details are not required in order to practice the invention. Thus, the foregoing descriptions of specific embodiments of the invention are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed; obviously, many modifications and embodiments are possible in view of the above teachings. The embodiments were chosen and described in order to explain the principles of the invention and its practical applications, they thereby enable others skilled in the art to utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the following claims and their equivalents define the scope of the invention.
Claims
What is claimed is:
1. A computer-implemented method for updating welding instructions for a welding robot, the computer-implemented method comprising:
receiving, via at least one sensor coupled to an arm of the welding robot, data of a workspace that includes a part to be welded, wherein the data is received while the welding robot welds the part according to the welding instructions;
transforming, via a processor, the data into a three-dimensional representation that comprises at least a portion of the part;
identifying, via the processor and based on recognition of one or more features in the three-dimensional representation by at least one neural network, a desired state indicative of a desired position of at least a portion of the welding robot with respect to the part;
comparing, via the processor, the desired state to an estimated state indicative of an estimated position of at least the portion of the welding robot with respect to the part; and
updating, via the processor and while the welding robot welds the part, the welding instructions based on the comparing.
2. The computer-implemented method of claim 1, wherein the estimated state is based at least in part on a model of the part.
3. The computer-implemented method of claim 2, wherein the model of the part includes annotations representing the welding instructions for the welding robot.
4. The computer-implemented method of claim 1, wherein the estimated state of the part is based at least in part on the data of the workspace.
5. The computer-implemented method of claim 1, wherein the at least one sensor includes a first sensor and a second sensor, wherein the data of the workspace comprises data from the first sensor and the second sensor, wherein the data from the first sensor shows the part and a welding tip of the welding robot from a different angle than the data from the second sensor, the computer-implemented method further comprising:
fusing, via the processor, data from the first sensor and data from the second sensor.
6. The computer-implemented method of claim 1, wherein updating the welding instructions based on the comparing comprises updating a motion of the welding robot to avoid a collision.
7. The computer-implemented method of claim 1, wherein the recognition comprises receiving, by the at least one neural network, at least the three-dimensional representation and then outputting an encoding of a classification of the one or more features in the three-dimensional representation as at least one of a tack weld, an edge, a gap, a hole, or a void.
8. The computer-implemented method of claim 7, wherein identifying the desired state is based on the encoding of the classification of the one or more features.
9. The computer-implemented method of claim 8, wherein identifying the desired state is based on comparing simulated three dimensional data with the data.
10. The computer-implemented method of claim 7, wherein updating the welding instructions occurs simultaneously with the identifying of the desired state, by the at least one neural network.
11. The computer-implemented method of claim 1, wherein updating the welding instructions includes adjusting a motion of the welding robot.
12. The computer-implemented method of claim 11, wherein updating the welding instructions comprises adjusting at least one of a position or an orientation of a motorized fixture in the workspace.
13. The computer-implemented method of claim 12, wherein updating the welding instructions comprises updating at least one of a welder voltage, a welder current, a duration of an electrical pulse, a shape of an electrical pulse, or a material feed rate.
14. The computer-implemented method of claim 7, wherein updating the welding instructions is based on the classification of the one or more features in the three-dimensional representation as the tack weld.
15. The computer-implemented method of claim 1, wherein updating the welding instructions comprises updating at least one of a welder voltage, a welder current, a duration of an electrical pulse, a shape of an electrical pulse, or a material feed rate.
16. A controller for a robotic welding system comprising a robotic arm and at least one sensor coupled to the robotic arm, the controller comprising:
a processor communicatively coupled to a non-transitory computer readable storage medium storing instructions that, when executed by the processor, cause actions including:
receiving, via at least one sensor coupled to an arm of the welding robot, data of a workspace that includes a part to be welded, wherein the data is received while the welding robot welds the part according to a welding instruction set;
transforming the data into a three-dimensional representation that comprises at least a portion of the part;
identifying, based on recognition of one or more features in the three-dimensional representation by at least one neural network, a desired state indicative of a desired position of at least a portion of the welding robot with respect to the part;
comparing the desired state to an estimated state indicative of an estimated position of at least the portion of the welding robot with respect to the part; and
updating, while the welding robot welds the part, the welding instruction set based on the comparing.
17. A robotic welding system, comprising:
a welding robot comprising an arm;
at least one sensor coupled to the arm; and
a controller comprising a processor communicatively coupled to a non-transitory computer readable storage medium storing instructions that, when executed by the processor, cause actions including:
receiving, via the at least one sensor, data of a workspace that includes a part to be welded, wherein the data is received while the welding robot welds the part according to a welding program;
transforming the data into a three-dimensional representation that comprises at least a portion of the part;
identifying, based on recognition of one or more features in the three-dimensional representation, of a first state indicative of a first position of at least a portion of the welding robot with respect to the part;
comparing the first state to a second state indicative of a second position of at least the portion of the welding robot with respect to the part, wherein the second state is based at least in part on the data of the workspace;
updating, while the welding robot welds the part, the welding instructions based on the comparing; and
welding the part with the welding robot according to the updated welding instructions.
18. The robotic welding system of claim 17, wherein the recognition of one or more features comprises receiving, by at least one neural network, at least the three-dimensional representation and then outputting an encoding of a classification of the one or more features in the three-dimensional representation as a tack weld, wherein updating the welding instructions is based on the classification.
19. The robotic welding system of claim 18, wherein updating the welding instructions is based on the classification of the one or more features in the three-dimensional representation as the tack weld, wherein updating the welding instructions comprises updating at least one of a welder voltage, a welder current, a duration of an electrical pulse, a shape of an electrical pulse, or a material feed rate.
20. The robotic welding system of claim 19, wherein updating the welding instructions further comprises updating a motion of the welding robot.
Patent Citations (254)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US1724301(A) | 1929-08-01 | Alma | Applicant |
| US3532807(A) | 1970-10-01 | Wall, Jr. et al. | Applicant |
| US4011437(A) | 1977-03-01 | Hohn | Applicant |
| US4021840(A) | 1977-05-01 | Ellsworth et al. | Applicant |
| US4148061(A) | 1979-04-01 | Lemelson | Applicant |
| US4234777(A) | 1980-11-01 | Balfanz | Applicant |
| US4255643(A) | 1981-03-01 | Balfanz | Applicant |
| US4380696(A) | 1983-04-01 | Masaki | Applicant |
| US4412121(A) | 1983-10-01 | Kremers et al. | Applicant |
| US4482968(A) | 1984-11-01 | Inaba et al. | Applicant |
| US4492847(A) | 1985-01-01 | Masaki et al. | Applicant |
| US4495588(A) | 1985-01-01 | Nio et al. | Applicant |
| US4497019(A) | 1985-01-01 | Waber | Applicant |
| US4497996(A) | 1985-02-01 | Libby, Jr. et al. | Applicant |
| US4515521(A) | 1985-05-01 | Takeo et al. | Applicant |
| US4553077(A) | 1985-11-01 | Brantmark et al. | Applicant |
| US4555613(A) | 1985-11-01 | Shulman | Applicant |
| US4561050(A) | 1985-12-01 | Iguchi et al. | Applicant |
| US4567348(A) | 1986-01-01 | Smith et al. | Applicant |
| US4568816(A) | 1986-02-01 | Casler, Jr. | Applicant |
| US4575304(A) | 1986-03-01 | Nakagawa et al. | Applicant |
| US4578554(A) | 1986-03-01 | Coulter | Applicant |
| US4580229(A) | 1986-04-01 | Koyama et al. | Applicant |
| US4587396(A) | 1986-05-01 | Rubin | Applicant |
| US4590577(A) | 1986-05-01 | Nio et al. | Applicant |
| US4593173(A) | 1986-06-01 | Bromley et al. | Applicant |
| US4594497(A) | 1986-06-01 | Takahashi et al. | Applicant |
| US4595989(A) | 1986-06-01 | Yasukawa et al. | Applicant |
| US4613942(A) | 1986-09-01 | Chen | Applicant |
| US4616121(A) | 1986-10-01 | Clocksin et al. | Applicant |
| US4617504(A) | 1986-10-01 | Detriche | Applicant |
| US4642752(A) | 1987-02-01 | Debarbieri et al. | Applicant |
| US4652803(A) | 1987-03-01 | Kamejima et al. | Applicant |
| US4675502(A) | 1987-06-01 | Haefner et al. | Applicant |
| US4677568(A) | 1987-06-01 | Arbter | Applicant |
| US4685862(A) | 1987-08-01 | Nagai et al. | Applicant |
| US4724301(A) | 1988-02-01 | Shibata et al. | Applicant |
| US4725965(A) | 1988-02-01 | Keenan | Applicant |
| US4744039(A) | 1988-05-01 | Suzuki et al. | Applicant |
| US4745857(A) | 1988-05-01 | Putnam et al. | Applicant |
| US4757180(A) | 1988-07-01 | Kainz et al. | Applicant |
| US4804860(A) | 1989-02-01 | Ross et al. | Applicant |
| US4812614(A) | 1989-03-01 | Wang et al. | Applicant |
| US4833383(A) | 1989-05-01 | Skarr et al. | Applicant |
| US4833624(A) | 1989-05-01 | Kuwahara et al. | Applicant |
| US4837487(A) | 1989-06-01 | Kurakake et al. | Applicant |
| US4845992(A) | 1989-07-01 | Dean | Applicant |
| US4899095(A) | 1990-02-01 | Kishi et al. | Applicant |
| US4906907(A) | 1990-03-01 | Tsuchihashi et al. | Applicant |
| US4907169(A) | 1990-03-01 | Lovoi | Applicant |
| US4924063(A) | 1990-05-01 | Buchel et al. | Applicant |
| US4945493(A) | 1990-07-01 | Huang et al. | Applicant |
| US4969108(A) | 1990-11-01 | Webb et al. | Applicant |
| US4973216(A) | 1990-11-01 | Domm | Applicant |
| US5001324(A) | 1991-03-01 | Aiello et al. | Applicant |
| US5006999(A) | 1991-04-01 | Kuno et al. | Applicant |
| US5053976(A) | 1991-10-01 | Nose et al. | Applicant |
| US5083073(A) | 1992-01-01 | Kato | Applicant |
| US5096353(A) | 1992-03-01 | Tesh et al. | Applicant |
| US5154717(A) | 1992-10-01 | Matsen, III et al. | Applicant |
| US5159745(A) | 1992-11-01 | Kato | Applicant |
| US5219264(A) | 1993-06-01 | McClure et al. | Applicant |
| US5245409(A) | 1993-09-01 | Tobar | Applicant |
| US5288991(A) | 1994-02-01 | King et al. | Applicant |
| US5300869(A) | 1994-04-01 | Skaar et al. | Applicant |
| US5326469(A) | 1994-07-01 | Thompson | Applicant |
| US5329469(A) | 1994-07-01 | Watanabe | Applicant |
| US5379721(A) | 1995-01-01 | Dessing et al. | Applicant |
| US5380978(A) | 1995-01-01 | Pryor | Applicant |
| US5457773(A) | 1995-10-01 | Jeon | Applicant |
| US5465037(A) | 1995-11-01 | Huissoon et al. | Applicant |
| US5479078(A) | 1995-12-01 | Karakama et al. | Applicant |
| US5511007(A) | 1996-04-01 | Nihei et al. | Applicant |
| US5532924(A) | 1996-07-01 | Hara et al. | Applicant |
| US5570458(A) | 1996-10-01 | Umeno et al. | Applicant |
| US5572102(A) | 1996-11-01 | Goodfellow et al. | Applicant |
| US5598345(A) | 1997-01-01 | Tokura | Applicant |
| US5600760(A) | 1997-02-01 | Pryor | Applicant |
| US5602967(A) | 1997-02-01 | Pryor | Applicant |
| US5608847(A) | 1997-03-01 | Pryor | Applicant |
| US5612785(A) | 1997-03-01 | Boillot et al. | Applicant |
| US5624588(A) | 1997-04-01 | Terawaki et al. | Applicant |
| US5828566(A) | 1998-10-01 | Pryor | Applicant |
| US5906761(A) | 1999-05-01 | Gilliland et al. | Applicant |
| US5925268(A) | 1999-07-01 | Britnell | Applicant |
| US5956417(A) | 1999-09-01 | Pryor | Applicant |
| US5959425(A) | 1999-09-01 | Bieman et al. | Applicant |
| US5961858(A) | 1999-10-01 | Britnell | Applicant |
| US6035695(A) | 2000-03-01 | Kim | Applicant |
| US6044308(A) | 2000-03-01 | Huissoon | Applicant |
| US6049059(A) | 2000-04-01 | Kim | Applicant |
| US6084203(A) | 2000-07-01 | Bonigen | Applicant |
| US6163946(A) | 2000-12-01 | Pryor | Applicant |
| US6167607(B1) | 2001-01-01 | Pryor | Applicant |
| US6304050(B1) | 2001-10-01 | Skaar et al. | Applicant |
| US6429404(B1) | 2002-08-01 | Suzuki | Applicant |
| US6430474(B1) | 2002-08-01 | DiStasio et al. | Applicant |
| US6804580(B1) | 2004-10-01 | Stoddard et al. | Applicant |
| US6909066(B2) | 2005-06-01 | Zheng et al. | Applicant |
| US7130718(B2) | 2006-10-01 | Gunnarsson et al. | Applicant |
| US7151848(B1) | 2006-12-01 | Watanabe et al. | Applicant |
| US7734358(B2) | 2010-06-01 | Watanabe et al. | Applicant |
| US7805219(B2) | 2010-09-01 | Ishikawa et al. | Applicant |
| US7813830(B2) | 2010-10-01 | Summers et al. | Applicant |
| US7818091(B2) | 2010-10-01 | Kazi et al. | Applicant |
| US7946439(B1) | 2011-05-01 | Toscano et al. | Applicant |
| US8494678(B2) | 2013-07-01 | Quandt et al. | Applicant |
| US8525070(B2) | 2013-09-01 | Tanaka et al. | Applicant |
| US8538125(B2) | 2013-09-01 | Linnenkohl et al. | Applicant |
| US8644984(B2) | 2014-02-01 | Nagatsuka et al. | Applicant |
| US9067321(B2) | 2015-06-01 | Landsnes | Applicant |
| US9221117(B2) | 2015-12-01 | Conrardy et al. | Applicant |
| US9221137(B2) | 2015-12-01 | Otts | Applicant |
| US9666160(B2) | 2017-05-01 | Patel et al. | Applicant |
| US9685099(B2) | 2017-06-01 | Boulware et al. | Applicant |
| US9724787(B2) | 2017-08-01 | Becker et al. | Applicant |
| US9773429(B2) | 2017-09-01 | Boulware et al. | Applicant |
| US9779635(B2) | 2017-10-01 | Zboray et al. | Applicant |
| US9799635(B2) | 2017-10-01 | Kuriki et al. | Applicant |
| US9802277(B2) | 2017-10-01 | Beatty et al. | Applicant |
| US9821415(B2) | 2017-11-01 | Rajagopalan et al. | Applicant |
| US9836987(B2) | 2017-12-01 | Postlethwaite et al. | Applicant |
| US9937577(B2) | 2018-04-01 | Daniel et al. | Applicant |
| US9975196(B2) | 2018-05-01 | Zhang et al. | Applicant |
| US9977242(B2) | 2018-05-01 | Patel et al. | Applicant |
| US9993891(B2) | 2018-06-01 | Wiryadinata | Applicant |
| US10040141(B2) | 2018-08-01 | Rajagopalan et al. | Applicant |
| US10083627(B2) | 2018-09-01 | Daniel et al. | Applicant |
| US10191470(B2) | 2019-01-01 | Inoue | Applicant |
| US10197987(B2) | 2019-02-01 | Battles et al. | Applicant |
| US10198962(B2) | 2019-02-01 | Postlethwaite et al. | Applicant |
| US10201868(B2) | 2019-02-01 | Dunahoo et al. | Applicant |
| US10551179(B2) | 2020-02-01 | Lonsberry et al. | Applicant |
| US11034024(B2) | 2021-06-01 | Saez et al. | Applicant |
| US11067965(B2) | 2021-07-01 | Spieker et al. | Applicant |
| US11179793(B2) | 2021-11-01 | Atherton et al. | Applicant |
| US11407110(B2) | 2022-08-01 | Lonsberry | Examiner |
| US11759952(B2) | 2023-09-01 | Lonsberry | Examiner |
| US12109709(B2) | 2024-10-01 | Lonsberry | Examiner |
| US12521884(B2) | 2026-01-01 | Schwenker | Examiner |
| US2001/0004718(A1) | 2001-06-01 | Gilliland et al. | Applicant |
| US2004/0105519(A1) | 2004-06-01 | Yamada et al. | Applicant |
| US2005/0023261(A1) | 2005-02-01 | Zheng et al. | Applicant |
| US2006/0047363(A1) | 2006-03-01 | Farrelly et al. | Applicant |
| US2006/0049153(A1) | 2006-03-01 | Cahoon et al. | Applicant |
| US2008/0114492(A1) | 2008-05-01 | Miegel et al. | Applicant |
| US2008/0125893(A1) | 2008-05-01 | Tilove et al. | Applicant |
| US2009/0075274(A1) | 2009-03-01 | Slepnev et al. | Applicant |
| US2009/0139968(A1) | 2009-06-01 | Hesse et al. | Applicant |
| US2010/0114338(A1) | 2010-05-01 | Bandyopadhyay et al. | Applicant |
| US2010/0152870(A1) | 2010-06-01 | Wanner et al. | Applicant |
| US2010/0206938(A1) | 2010-08-01 | Quandt et al. | Applicant |
| US2010/0274390(A1) | 2010-10-01 | Walser et al. | Applicant |
| US2010/0326962(A1) | 2010-12-01 | Calla | Examiner |
| US2011/0141251(A1) | 2011-06-01 | Marks et al. | Applicant |
| US2011/0282492(A1) | 2011-11-01 | Krause | Examiner |
| US2011/0297666(A1) | 2011-12-01 | Ihle et al. | Applicant |
| US2012/0096702(A1) | 2012-04-01 | Kingsley et al. | Applicant |
| US2012/0145771(A1) | 2012-06-01 | Bohlin | Examiner |
| US2012/0267349(A1) | 2012-10-01 | Berndl et al. | Applicant |
| US2013/0119040(A1) | 2013-05-01 | Suraba et al. | Applicant |
| US2013/0123801(A1) | 2013-05-01 | Umasuthan et al. | Applicant |
| US2013/0259376(A1) | 2013-10-01 | Louban | Applicant |
| US2014/0088577(A1) | 2014-03-01 | Anastassiou et al. | Applicant |
| US2014/0100694(A1) | 2014-04-01 | Rueckl et al. | Applicant |
| US2014/0309762(A1) | 2014-10-01 | Hayata | Examiner |
| US2014/0365009(A1) | 2014-12-01 | Wettels | Applicant |
| US2015/0122781(A1) | 2015-05-01 | Albrecht | Applicant |
| US2015/0127162(A1) | 2015-05-01 | Gotou | Applicant |
| US2016/0016261(A1) | 2016-01-01 | Mudd | Examiner |
| US2016/0096269(A1) | 2016-04-01 | Atohira et al. | Applicant |
| US2016/0125592(A1) | 2016-05-01 | Becker et al. | Applicant |
| US2016/0125593(A1) | 2016-05-01 | Becker et al. | Applicant |
| US2016/0224012(A1) | 2016-08-01 | Hunt | Applicant |
| US2016/0257000(A1) | 2016-09-01 | Guerin et al. | Applicant |
| US2016/0257070(A1) | 2016-09-01 | Boydston et al. | Applicant |
| US2016/0267636(A1) | 2016-09-01 | Duplaix et al. | Applicant |
| US2016/0267806(A1) | 2016-09-01 | Hsu et al. | Applicant |
| US2017/0028499(A1) | 2017-02-01 | Yoshida et al. | Applicant |
| US2017/0072507(A1) | 2017-03-01 | Legault | Applicant |
| US2017/0132807(A1) | 2017-05-01 | Shivaram et al. | Applicant |
| US2017/0232615(A1) | 2017-08-01 | Hammock et al. | Applicant |
| US2017/0266758(A1) | 2017-09-01 | Fukui et al. | Applicant |
| US2017/0364076(A1) | 2017-12-01 | Keshmiri et al. | Applicant |
| US2017/0368649(A1) | 2017-12-01 | Marrocco et al. | Applicant |
| US2018/0043471(A1) | 2018-02-01 | Aoki | Applicant |
| US2018/0065204(A1) | 2018-03-01 | Burrows | Applicant |
| US2018/0117701(A1) | 2018-05-01 | Ge et al. | Applicant |
| US2018/0147662(A1) | 2018-05-01 | Furuya | Applicant |
| US2018/0266961(A1) | 2018-09-01 | Narayanan et al. | Applicant |
| US2018/0266967(A1) | 2018-09-01 | Ohno et al. | Applicant |
| US2018/0304550(A1) | 2018-10-01 | Atherton et al. | Applicant |
| US2018/0322623(A1) | 2018-11-01 | Memo et al. | Applicant |
| US2018/0341730(A1) | 2018-11-01 | Atherton et al. | Applicant |
| US2019/0076949(A1) | 2019-03-01 | Atherton et al. | Applicant |
| US2019/0108639(A1) | 2019-04-01 | Tchapmi et al. | Applicant |
| US2019/0178817(A1) | 2019-06-01 | Roux et al. | Applicant |
| US2019/0193180(A1) | 2019-06-01 | Troyer et al. | Applicant |
| US2019/0240759(A1) | 2019-08-01 | Ennsbrunner et al. | Applicant |
| US2019/0375101(A1) | 2019-12-01 | Miyata et al. | Applicant |
| US2020/0021780(A1) | 2020-01-01 | Jeong et al. | Applicant |
| US2020/0030984(A1) | 2020-01-01 | Suzuki et al. | Applicant |
| US2020/0114449(A1) | 2020-04-01 | Chang et al. | Applicant |
| US2020/0130089(A1) | 2020-04-01 | Ivkovich et al. | Applicant |
| US2020/0139471(A1) | 2020-05-01 | Pliska et al. | Applicant |
| US2020/0164521(A1) | 2020-05-01 | Li et al. | Applicant |
| US2020/0180062(A1) | 2020-06-01 | Suzuki et al. | Applicant |
| US2020/0223064(A1) | 2020-07-01 | Zak | Applicant |
| US2020/0262079(A1) | 2020-08-01 | Saez et al. | Applicant |
| US2020/0269340(A1) | 2020-08-01 | Tang et al. | Applicant |
| US2020/0316779(A1) | 2020-10-01 | Truebenbach et al. | Applicant |
| US2020/0409376(A1) | 2020-12-01 | Ebrahimi Afrouzi et al. | Applicant |
| US2021/0012678(A1) | 2021-01-01 | Torrecilla et al. | Applicant |
| US2021/0138646(A1) | 2021-05-01 | Matsushima | Examiner |
| US2021/0158724(A1) | 2021-05-01 | Becker et al. | Applicant |
| US2021/0318673(A1) | 2021-10-01 | Kitchen et al. | Applicant |
| US2022/0016776(A1) | 2022-01-01 | Lonsberry | Examiner |
| US2022/0226922(A1) | 2022-07-01 | Albrecht et al. | Applicant |
| US2022/0250183(A1) | 2022-08-01 | Knoener | Applicant |
| US2022/0258267(A1) | 2022-08-01 | Becker | Applicant |
| US2022/0266453(A1) | 2022-08-01 | Lonsberry et al. | Applicant |
| US2022/0305593(A1) | 2022-09-01 | Lonsberry et al. | Applicant |
| US2022/0324110(A1) | 2022-10-01 | Lonsberry | Examiner |
| US2022/0402147(A1) | 2022-12-01 | Martin et al. | Applicant |
| US2022/0410402(A1) | 2022-12-01 | Lonsberry | Examiner |
| US2023/0093558(A1) | 2023-03-01 | Takeya et al. | Applicant |
| US2023/0123712(A1) | 2023-04-01 | Lonsberry | Examiner |
| US2023/0143710(A1) | 2023-05-01 | Ikeguchi | Examiner |
| US2023/0173676(A1) | 2023-06-01 | Lonsberry | Examiner |
| US2023/0222930(A1) | 2023-07-01 | Shiomi | Examiner |
| US2023/0260138(A1) | 2023-08-01 | Becker et al. | Applicant |
| US2023/0278224(A1) | 2023-09-01 | Bunker et al. | Applicant |
| US2023/0330764(A1) | 2023-10-01 | Ott et al. | Applicant |
| US2023/0403475(A1) | 2023-12-01 | Huang | Examiner |
| US2024/0025041(A1) | 2024-01-01 | Lonsberry | Examiner |
| US2024/0100633(A1) | 2024-03-01 | Fu | Examiner |
| US2024/0416519(A1) | 2024-12-01 | Lonsberry | Examiner |
| US2024/0424611(A1) | 2024-12-01 | Ogata | Examiner |
| US2025/0153261(A1) | 2025-05-01 | Vasu et al. | Applicant |
| CN110064818(A) | 2019-07-01 | Applicant | |
| CN110270997(A) | 2019-09-01 | Applicant | |
| CN110524581(A) | 2019-12-01 | Applicant | |
| CN111189393(A) | 2020-05-01 | Applicant | |
| CN113352317(A) | 2021-09-01 | Applicant | |
| EP3812105(A1) | 2021-04-01 | Applicant | |
| JP2010269336(A) | 2010-12-01 | Applicant | |
| JP2015140171(A) | 2015-08-01 | Applicant | |
| JP2015223604(A) | 2015-12-01 | Applicant | |
| JP2017196653(A) | 2017-11-01 | Applicant | |
| WO1994003303 | 1994-02-01 | Applicant | |
| WO2016013171(A1) | 2016-01-01 | Applicant | |
| WO2017115015(A1) | 2017-07-01 | Applicant | |
| WO2018173655(A1) | 2018-09-01 | Applicant | |
| WO2019153090(A1) | 2019-08-01 | Applicant |
Non-Patent Literature (40)
- Ahmed, S.M. (Oct. 2016) “Object Detection and Motion Planning for Automated Welding of Tubular Joints” 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, Daejeon, Korea, Oct. 9-14; pp. 2610-2615.Applicant
- Bingul et al., “A real-time prediction model of electrode extension for GMAW”, IEEE/ASME Transaction on Mechatronics, vol. 11, No. 1,pp. 47-4, 2006.Applicant
- Boulware, “A methodology for the robust procedure development of fillet welds”, The Ohio State University Knowledge Bank, 2006.Applicant
- Chandrasekaran, “Predication of bead geometry in gas metal arc welding by statistical regression analysis”, University of Waterloo, 2019.Applicant
- Devaraj et al., “Grey-based Taguchi multiobjective optimization and artificial intelligence-based prediction of dissimilar gas metal arc welding process performance”, Metals, vol. 11, 1858, 2021.Applicant
- Ding et al., A multi-bead overlapping model for robotic wire and arc additive manufacturing (WAAM), Robotics and Computer-Integrated Manufacturing, vol. 31, pp. 101-110, 2015.Applicant
- Ding, et al., “Bead modelling and implementation of adaptive MAT path in wire and arc additive manufacturing”, Robotics and Computer-Integrated Manufacturing, vol. 39, pp. 32-42, 2016.Applicant
- Hao et al., “Effects of tilt angle between laser nozzle and substrate on bead morphology in multi-axis laser cladding”, Journal of manufacturing processes, vol. 43, pp. 311-322, 2019.Applicant
- Harwig, “A wise method for assessing arc welding performance and quality”, Welding Journal, pp. 35-39, 2000.Applicant
- Harwig, “Arc behaviour and metal transfer of the vp-gmaw process”, School of Industrial and Manufacturing Science, 2003.Applicant
- He et al., “Dynamic modeling of weld bead geometry features in thick plate GMAW based on machine vision and learning”, Sensors, vol. 20, 7104, 2020.Applicant
- Horvath et al., “Bead geometry modeling on uneven base metal surface by fuzzy C1 systems for multi-pass welding”, Expert Systems with Applications, 186, 115356, 24 pages, 2021.Applicant
- Hu et al., “Molten pool behaviors and forming appearance of robotic GMAW on complex surface with various welding positions”, Journal of Manufacturing Processes, vol. 64, pp. 1359-1376, 2021.Applicant
- Hu et al., “Multi-bead overlapping model with varying cross-section profile for robotic GMAW-based additive manufacturing”, Journal of Intelligent Manufacturing, 2019.Applicant
- International Application No. PCT/US2021/042218, filed Jul. 19, 2021, by Path Robotics, Inc.: International Search Report and Written Opinion, mailed Dec. 9, 2021; 11 pages.Applicant
- Jing et al., “Rgb-d sensor-based auto path generation method for arc welding robot”, 2016 Chinese Control and Decision Conference (CCDC), IEEE, 2016.Applicant
- Kesse et al., “Development of an artificial intelligence powered TIG welding algorithm for the prediction of bead geometry for TIG welding processes using hybrid deep learning”, Metals, vol. 10, 451, 2020.Applicant
- Kiran et al., “Arc interaction and molten pool behavior in the three wire submerged arc welding process”, International Journal of Heat and Mass Transfer, vol. 87, pp. 327-340, 2015.Applicant
- Kolahan et al., A new approach for predicting and optimizing weld bead geometry in GMAW, World Academy of Science, Engineering and Technology, vol. 59, pp. 138-141, 2009.Applicant
- Larkin et al., “Automatic program generation for welding robots from CAD”, IEEE International Conference on Advanced Intelligent Mechatronics(AIM), IEEE, pp. 560-565, 2016.Applicant
- Li et al., “Enhanced beads overlapping model for wire and arc additive manufacturing of multi-layer multi-bead metallic parts”, Journal of Materials Processing Technology, 2017.Applicant
- Li et al., “GMAW-based additive manufacturing of inclined multi-layer multi-bead parts with flat-position deposition”, Journal of Materials Processing Tech, vol. 262, pp. 359-371, 2018.Applicant
- Liu et al., “Motion navigation for arc welding robots based on feature mapping in a simulation environment”, Robotics and Computer-Integrated Manufacturing, 26(2); pp. 137-144, 2010.Applicant
- Lv et al., “Levenberg-marquardt backpropagation training of multilayer neural networks for state estimation of a safety critical cyber-physical system”, 2018.Applicant
- Lynch, K.M. et al., “Robot Control”, Modern Robotics: Mechinics, Planning, and Control, Cambridge University Press, 2017, 403-460.Applicant
- Martinez et al., “Two gas metal arc welding process dataset of arc parameters and input parameters”, Data in Brief, vol. 35, 106790, 2021.Applicant
- Mollayi et al., “Application of multiple kernel support vector regression for weld bead geometry prediction in robotic GMAW Process”, International Journal of Electrical and Computer Engineering, vol. 8, No. pp. 2310-2318, 2018.Applicant
- Moos, et al., “Resistance spot welding process simulation for variational analysis on copmliant assemblies”, Journal of Manufacturing Systems, Dec. 16, pp. 44-71, 2014.Applicant
- Munro, “An empirical model for overlapping bead geometry during laser consolidation”, Defence Research and Development Canada, 2019.Applicant
- Murray, “Selecting parameters for GMAW using dimensional analysis”, Welding Research, pp. 152s-131s, 2002.Applicant
- Natarajan, S. et al., “Aiding Grasp Synthesis for Novel Objects Using Heuristic-Based and Data-Driven Active Vision Methods”, Frontiers in Robotics and AI, 8:696587, 2021.Applicant
- Nguyen et al., “Multi-bead overlapping models for tool path generation in wire-arc additive manufacturing processes”, Procedia Manufacturing, vol. 47, pp. 1123-1128, 2020.Applicant
- Rao et al., “Effect of process parameters and mathematical model for the prediction of bead geometry in pulsed GMA welding”, The International Journal of Advanced Manufacturing Technology, vol. 45, pp. 496-505, 2009.Applicant
- Ravichandran et al., “Parameter optimization of gas metal arc welding process on AISI: 430Stainless steel using meta heuristic optimization techniques”, Department of Mechatronics Engineering.Applicant
- Sheng et al., “A new adaptive trust region algorithm for optimization problems”, Acta Mathematica Scientia, vol. 38b, No. 2., pp. 479-496, 2018.Applicant
- Suryakumar et al., “Weld bead modeling and process optimization in hybrid layered manufacturing”, Computer-Aided Design, vol. 43, pp. 331-344, 211, 2011.Applicant
- Thao, et al., “Interaction model for predicting bead geometry for lab joint in GMA welding process”, Computational Materials Science and Surface Engineering, vol. 1, issue. 4., pp. 237-244, 2009.Applicant
- Wang, Xuewu, et al. “Welding robot collision-free path optimization.” Applied Sciences 7.2 (2017): 89. (Year: 2017).Applicant
- Xiao, R. et al., “An adaptive feature extraction algorithm for multiple typical seam tracking based on vision sensor in robotic arc welding” Sensors and Actuators A: Physical, 297:111533, 15 pages, 2019.Applicant
- Xiong et al., “Modeling of bead section profile and overlapping beads with experimental validation for robotic GMAW-based rapid manufacturing”, Robotics and Computer-Integrated Manufacturing, vol. 29, pp. 417-423, 2013.Applicant