System and method for selecting an item from a plurality of identified items by filtering out back images of the items

[0001] This application is a continuation-in-part of U.S. patent application Ser. No. 18/366,155 filed on Aug. 7, 2023, entitled “SYSTEM AND METHOD FOR IDENTIFYING A SECOND ITEM BASED ON AN ASSOCIATION WITH A FIRST ITEM”, which is a continuation-in-part of U.S. patent application Ser. No. 17/455,903 filed on Nov. 19, 2021, entitled “ITEM LOCATION DETECTION USING HOMOGRAPHIES,” which is a continuation-in-part of U.S. patent application Ser. No. 17/362,261 filed Jun. 29, 2021, entitled “ITEM IDENTIFICATION USING DIGITAL IMAGE PROCESSING,” which are all incorporated herein by reference.

[0002] The present disclosure relates generally to digital image processing, and more specifically to a system and method for selecting an item from a plurality of identified items by filtering out back images of the items.

[0003] Identifying and tracking objects within a space poses several technical challenges. For example, identifying different features of an item that can be used to later identify the item in an image is computationally intensive when the image includes several items. This process may involve identifying an individual item within the image and then comparing the features for an item against every item in a database that may contain thousands of items. In addition to being computationally intensive, this process requires a significant amount of time which means that this process is not compatible with real-time applications. This problem becomes intractable when trying to simultaneously identify and track multiple items.

[0004] The system disclosed in the present application provides a technical solution to the technical problems discussed above by using a combination of cameras and three-dimensional (3D) sensors to identify and track items that are placed on a platform. The disclosed system provides several practical applications and technical advantages which include a process for selecting a combination of cameras on an imaging device to capture images of items that are placed on a platform, identifying the items that are placed on the platform, and assigning the items to a user. Requiring a user to scan or manually identify items creates a bottleneck in the system's ability to quickly identify items. In contrast, the disclosed process is able to identify items from images of the items and assign the items to a user without requiring the user to scan or otherwise identify the items. This process provides a practical application of image detection and tracking by improving the system's ability to quickly identify multiple items. These practical applications not only improve the system's ability to identify items but also improve the underlying network and the devices within the network. For example, this disclosed process allows the system to service a larger number of users by reducing the amount of time that it takes to identify items and assign items to a user, while improving the throughput of image detection processing. In other words, this process improves hardware utilization without requiring additional hardware resources which increases the number of hardware resources that are available for other processes and increases the throughput of the system. Additionally, these technical improvements allow for scaling of the item identification and tracking functionality described herein.

[0005] In one embodiment, the item tracking system comprises an item tracking device that is configured to detect a triggering event at a platform of an imaging device. The triggering event may correspond with when a user approaches or interacts with the imaging device by placing items on the platform. The item tracking device is configured to capture a depth image of items on the platform using a 3D sensor and to determine an object pose for each item on the platform based on the depth image. The pose corresponds with the location and the orientation of an item with respect to the platform. The item tracking device is further configured to identify one or more cameras from among a plurality of cameras on the imaging device based on the object pose for each item on the platform. This process allows the item tracking device to select the cameras with the best views of the items on the platform which reduces the number of images that are processed to identify the items. The item tracking device is further configured to capture images of the items on the platform using the identified cameras and to identify the items within the images based on features of the items. The item tracking device is further configured to identify a user associated with the identified items on the platform, to identify an account that is associated with the user, and to add the items to the account that is associated with the user.

[0006] In another embodiment, the item tracking system comprises an item tracking device that is configured to capture a first overhead depth image of the platform using a 3D sensor at a first time instance and a second overhead depth image of a first object using the 3D sensor at a second time instance. The item tracking device is further configured to determine that a first portion of the first object is within a region-of-interest and a second portion of the first object is outside the region-of-interest in the second overhead depth image. The item tracking device is further configured to capture a third overhead depth image of a second object placed on the platform using the 3D sensor at a third time instance. The item tracking device is further configured to capture a first image of the second object using a camera in response to determining that the first object is outside of the region-of-interest and the second object is within the region-of-interest for the platform.

[0007] In another embodiment, the item tracking system comprises an item tracking device that is configured to identify a first pixel location within a first plurality of pixels corresponding with an item in a first image and to apply a first homography to the first pixel location to determine a first (x,y) coordinate. The item tracking device is further configured to identify a second pixel location within a second plurality of pixels corresponding with the item in a second image and to apply a second homography to the second pixel location to determine a second (x,y) coordinate. The item tracking device is further configured to determine that the distance between the first (x,y) coordinate and the second (x,y) coordinate is less than or equal to the distance threshold value, to associate the first plurality of pixels and the second plurality of pixels with a cluster for the item, and to output the first plurality of pixels and the second plurality of pixels.

[0008] In another embodiment, the item tracking system comprises an item tracking device that is configured to detect a triggering event corresponding with a user placing a first item on the platform, to capture a first image of the first item on the platform using a camera, and to input the first image into a machine learning model that is configured to output a first encoded vector based on features of the first item that are present in the first image. The item tracking device is further configured to identify a second encoded vector in an encoded vector library that most closely matches the first encoded vector and to identify a first item identifier in the encoded vector library that is associated with the second encoded vector. The item tracking device is further configured to identify the user, to identify an account that is associated with the user, and to associate the first item identifier with the account of the user.

[0009] In another embodiment, the item tracking system comprises an item tracking device that is configured to receive a first encoded vector and receive one or more feature descriptors for a first object. The item tracking device is further configured to remove one or more encoded vectors from an encoded vector library that are not associated with the one or more feature descriptors and to identify a second encoded vector in the encoded vector library that most closely matches the first encoded vector based on the numerical values within the first encoded vector. The item tracking device is further configured to identify a first item identifier in the encoded vector library that is associated with the second encoded vector and to output the first item identifier.

[0010] In another embodiment, the item tracking system comprises an item tracking device that is configured to capture a first image of an item on a platform using a camera and to determine a first number of pixels in the first image that corresponds with the item. The item tracking device is further configured to capture a first depth image of an item on the platform using a three-dimensional (3D) sensor and to determine a second number of pixels within the first depth image that corresponds with the item. The item tracking device is further configured to determine that the difference between the first number of pixels in the first image and the second number of pixels in the first depth image is less than the difference threshold value, to extract the plurality of pixels corresponding with the item in the first image from the first image to generate a second image, and to output the second image.

[0011] In another embodiment, the item tracking system comprises an item tracking device that is configured to receive a first point cloud data for a first item, to identify a first plurality of data points for the first object within the first point cloud data, and to extract the first plurality of data points from the first point cloud data. The item tracking device is further configured to receive a second point cloud data for the first item, to identify a second plurality of data points for the first object within the second point cloud data, and to extract a second plurality of data points from the second point cloud data. The item tracking device is further configured to merge the first plurality of data points and the second plurality of data points to generate combined point cloud data and to determine dimensions for the first object based on the combined point cloud data.

[0012] The disclosed system further contemplates an unconventional system and method for camera re-calibration based on an updated homography. More specifically, the disclosed system provides the practical application and technical improvements to the item identification and tracking techniques by detecting that an initial homography is no longer accurate, and in response, generate a new homography and re-calibrate the cameras and/or 3D sensors using the new homography.

[0013] In current approaches, when cameras and 3D sensors are deployed on an imaging device, the cameras and 3D sensors may be calibrated during an initial calibration process so pixel locations in an image captured by a given camera/3D sensor is mapped to respective physical location on the platform in the global plane. For example, during the initial calibration of cameras, a paper printed with unique patterns of checkboards may be placed on the platform. Each camera may capture an image of the paper and transmit to the item tracking device. The item tracking device may generate the homography that maps pixel locations of each unique pattern on the paper shown in the image to corresponding physical locations of unique pattern on the paper that is placed on the platform. Similar operations may be performed with respect to depth images captured by the 3D sensor.

[0014] After the initial camera calibration process, the item tracking engine may determine the physical location of any item placed on the platform by applying the homography to the pixel locations of the item shown in an image of the item. In some cases, a camera, 3D sensor, and/or the platform may move or be shifted from its initial location due to any number of reasons, such as an impact from a person when the person places an item on the platform, and the like. Because the initial homograph is determined based on the initial locations of the camera, 3D sensor, and the platform, a change in the initial location of one or more of the camera, 3D sensor, and the platform may lead to the homography becoming inaccurate. As a result, applying the homography to subsequent pixel locations of items shown in images or depth images may not lead to the actual physical location of the items on the platform.

[0015] In practice, it is very difficult, if not impossible, to know if a camera, 3D sensor, and/or the platform is shifted in position if no one witnessed it or it is captured on a camera facing the imaging device. One potential solution to this problem of a camera, 3D sensor and/or platform being shifted resulting in an inaccurate homography is to provide a routine maintenance to the cameras, 3D sensor, and platform to ensure that they are not shifted from their respective original locations. However, this potential solution is not feasible given that the imaging device may be deployed in a store and routine maintenance of the cameras, 3D sensor, and platform will interrupt the item check-out process. Besides, routine maintenance is labor-intensive and requires precise measurement of locations of the cameras, 3D sensor, and platform, which makes it an error-prone process.

[0016] The present disclosure provides a solution to this and other technical problems that are currently arising in the realm of item identification and tracking technology. For example, the present system is configured to detect if there is shift in location of any of the camera, 3D sensor, and/or platform, and in response to detecting the shift in location of any of the camera, 3D sensor, and platform, generate a new homography, and re-calibrate the camera, 3D sensor using the new homography. In this manner, the disclosed system improves the item identifying and tracking techniques. For example, the disclosed system increases the accuracy in item tracking and identification techniques, specifically, in cases where a camera, a 3D sensor, and/or platform has moved from its initial position when the initial homography was generated and determined. Accordingly, the disclosed system provides the practical application and technical improvements to the item identification and tracking techniques. For example, the disclosed system offers technical improvements in the field of item identification and tracking technology by addressing the inherent challenge of maintaining accuracy in a dynamic environment. For example, the disclosed system continuously or periodically (e.g., every second, every milliseconds, etc.) may monitor the positions of cameras, 3D sensors, and the platform. When the disclosed system detects any shift in the location of any of these components, the disclosed system generates a new homography and recalibrates the cameras and 3D sensors accordingly. Therefore, the pixel-to-physical location mapping remains precise (or within an acceptable precision threshold), even in scenarios where the system components have been moved or shifted. Furthermore, the disclosed system increases reliability by proactively addressing challenges of shifts in locations of cameras, 3D sensors, and the platform and maintains high accuracy even in changing conditions. In this manner, the disclosed system provides additional practical applications and technical improvements to the item identification and tracking technology. Accordingly, this represents an improvement to the efficiency, throughput, and productivity of computer systems implemented to perform the described operations.

[0017] In some embodiments, an object tracking system comprises a plurality of cameras, a memory, and a processor. Each camera is configured to capture images of at least a portion of a platform. The memory is configured to store a first homography that is configured to translate between pixel locations in an image and physical (x,y) coordinates in a global plane. The memory is further configured to store a reference location array comprising a first set of physical (x,y) locations of a set of points located on a calibration board in the global plane. Each of the first set of physical (x,y) locations is associated with a point from the set of points. The calibration board is positioned on the platform. The reference location array is determined by the first homography. The processor is communicatively coupled with the plurality of cameras and the memory. The processor is configured to receive a first image from a first camera, wherein the first image shows at least a portion of the set of points on the calibration board. The processor is further configured to determine a first pixel location array that comprise a first set of pixel locations associated with the set of points in the first image. The processor is further configured to determine, by applying the first homography to the first pixel location array, a first calculated location array identifying a first set of calculated physical (x,y) location coordinates of the set of points in the global plane. The processor is further configured to compare the reference location array with the first calculated location array. The processor is further configured to determine a difference between the reference location array and the first calculated location array. The processor is further configured to determine that the difference between the reference location array and the first calculated location array is more than a threshold value. In response to determining that the difference between the reference location array and the first calculated location array is more than the threshold value, the processor is further configured to determine that the first camera and/or the platform has moved from a respective initial location when the first homography was determined. The processor is further configured to determine a second homography by multiplying an inverse of the first pixel location array with by the reference location array. The processor is further configured to calibrate the first camera using the second homography.

[0018] Certain embodiments of the present disclosure describe techniques for detecting a triggering event corresponding to a placement of an item on a platform of an imaging device. An overhead camera positioned above the platform and having a top view of the platform is configured to take pictures of the platform (e.g., periodically or continually). Each particular pixel of an image captured by the overhead camera is associated with a depth value indicative of a distance between the overhead camera and a surface depicted by the particular pixel. A reference image of an empty platform is captured and an average reference depth value associated with all pixels in the reference image is calculated. Thereafter, for each subsequent image captured by the overhead camera, a real-time average depth associated with all pixels of the subsequent image is calculated and subtracted from the reference depth calculated for the empty platform. When the difference between the reference depth and real-time depth stays constant above zero across several images of the platform, it means that an item has been placed on the platform and is ready for identification. In response, a triggering event is determined to have been detected.

[0019] The system and method described in these embodiments of the present disclosure provide a practical application of intelligently detecting a triggering event corresponding to placement of an item on the platform of the imaging device. As described with reference to FIGS. 32A-B, 33A-D, and 34, an item tracking device detects whether an item has been placed on the platform by comparing a reference overhead image of an empty platform with a plurality of subsequently captured overhead images of the platform. By calculating a difference in the average depth values associated with pixels of the reference image and the plurality of subsequent images, the item tracking device determines, for example, that a user's hand holding an item entered the platform, placed the first item on the platform, and exited the platform. This technique for detecting a triggering event avoids false detection of triggering events as well as avoids missed detection of triggering events, thus improving accuracy associated with detecting triggering events at the platform. Further, by avoiding false detection of triggering events, the disclosed system and method saves computing resources (e.g., processing and memory resources associated with the item tracking device) which would otherwise be used to perform one or more processing steps that follow the detection of a triggering event such as capturing images using cameras of the imaging device to identify items placed on the platform. This, for example, improves the processing efficiency associated with the processor of the item tracking device. Thus, the disclosed system and method generally improve the technology associated with automatic detection of items.

[0020] Certain embodiments of the present disclosure describe techniques for detecting an item that was placed on a platform of an imaging device in a previous interaction and assigning to the item an item identifier that was identified in the previous interaction. The disclosed techniques determine whether an item has moved on the platform between interactions associated with a particular transaction. Upon determining that the item has not moved between interactions, the item is assigned an item identifier that was identified as part of a previous interaction. For example, when a first item is placed on the platform for the first time as part of an interaction, a first image of the first item is captured using an overhead camera positioned above the platform. An item identifier is determined for the first item and stored in a memory. Subsequently, when a second item is placed on the platform as part of a subsequent interaction, a second image of the first item is captured using the overhead camera. The second image of the first item is compared with the first image of the first item. When an overlap between the first and second images of the first item equals or exceeds a threshold, it is determined that the first item has not moved from its position on the platform between the first and second interactions. In response to determining that the first item has not moved between the two interactions, the first item is assigned the item identifier that was identified as part of the first interaction.

[0021] The system and method described in these embodiments of the present disclosure provide a practical application of intelligently determining whether an item has moved on the platform between interactions and assigning a previously identified item identifier to the item in response to determining that the item has not moved on the platform between interactions. As described with reference to FIGS. 35A-B, 36A-B, and 37, an item tracking device determines whether an item has moved between two interactions by comparing overhead images of the item captured during the two interactions. When an overlap between the overhead images equals or exceeds a threshold, the item tracking device determines that the item has not moved on the platform between the two interactions, and in response, assigns an item identifier to the item that was identified in a previous interaction. These techniques save computing resources (e.g., processing and memory resources associated with the item tracking device) that would otherwise be used to re-run item identification algorithms for items that were already identified as part of a previous interaction. This, for example, improves the processing efficiency associated with the processor of the item tracking device. Thus, the disclosed system and method generally improve the technology associated with automatic detection of items.

[0022] Certain embodiments of the present disclosure describe techniques for detecting an item that was placed on a platform of an imaging device in a previous interaction and assigning to the item an item identifier that was identified in the previous interaction. The disclosed techniques may detect an item that has moved on the platform between interactions associated with a transaction. Upon detecting an item from a previous interaction that may have moved on the platform between interactions, the item is assigned an item identifier that was identified as part of a previous interaction. For example, when a first item is placed on the platform for the first time as part of an interaction, a plurality of first images of the first item are captured using a plurality of cameras associated with the imaging device. The item is identified based on the plurality of first images of the first item. Subsequently, when a second item is placed on the platform as part of a subsequent interaction, a plurality of second images of the first item are captured using the same cameras. Each first image of the first item captured using a particular camera is compared with a second image of the first item captured using the same camera. When a majority of the first images match with the corresponding second images of the first item, it is determined that the second images correspond to the first item and, in response, the first item is assigned the item identifier that was identified as part of the first interaction.

[0023] The system and method described these embodiments of the present disclosure provide a practical application of intelligently identifying an item that was placed on the platform of the imaging device as part of a previous interaction and assigning the item an item identifier that was identified for the item in the previous interaction. As described with reference to FIGS. 38A-B, and 39A-B, in response to detecting that the first item has been placed on the platform as part of the first interaction, an item tracking device captures a plurality of first images of the first item, generates a plurality of cropped first images of the first item based on the first images, identifies the first item based on the cropped first images, and stores a first item identifier associated with the first item in a memory. In response to detecting that a second item has been added on the platform as part of a second interaction, item tracking device captures a plurality of second images of the first item and generates a plurality of cropped second images of the first item based on the second images. Item tracking device compares the cropped first images with the cropped second images. When item tracking device determines that the cropped first images match with the cropped second images, item tracking device determines that the cropped second images are associated with (e.g., depict) the first item that was identified as part of the first interaction. In response, item tracking device accesses the first item identifier from the memory and assigns the first item identifier to the first item. These techniques save computing resources (e.g., processing and memory resources associated with the item tracking device) that would otherwise be used to re-run item identification algorithms for items that were already identified as part of a previous interaction. This, for example, improves the processing efficiency associated with the processor of the item tracking device. Thus, the disclosed system and method generally improve the technology associated with automatic detection of items.

[0024] The disclosed system further contemplates an unconventional system and method for item identification using container-based classification. More specifically, the disclosed system provides practical applications and technical improvements to the item identification and tracking techniques by reducing the search set to a subset of items that are associated with a container category of an item in question.

[0025] In some cases, the same container, such as a cup, a box, a bottle, and the like, may be used for multiple items. For example, in some cases, a user may pour an item (e.g., tea, soda) into a container that is designated for another item (e.g., in a coffee cup) and place the container on the platform. In such cases, it is challenging to recognize what item is actually placed inside the container and it would require a large amount of computing resources and training data to recognize the item.

[0026] The present disclosure provides a solution to this and other technical problems that are currently arising in the realm of item identification and tracking technology. For example, the disclosed system is configured to associate each item with one or more container categories that have been used by users to place the item into. Thus, the disclosed system generates groups of container categories and classifies each item into an appropriate container category.

[0027] During the item identification process for an item, the disclosed system determines a container category associated with the item, identifies items that belong to the same class of container category as the item, and present the identified items in a list of item options on a graphical user interface (GUI) for the user to choose from. The user may select an item from the list on the GUI. The disclosed system uses the user selection as feedback in the item identification process. In this manner, the disclosed system improves the item identifying and tracking techniques. For example, the disclosed system may reduce the search space dataset from among the encoded vector library that includes encoded feature vectors representing all the items available at the physical location (e.g., store) to a subset of entries that are associated with the particular container category that is associated with the item in question.

[0028] By reducing the search space dataset to a subset that is associated with the particular container category as the item in question, the item tracking device does not have to consider the rest of the items that are not associated with the particular container category. Therefore, the disclosed system provides a practical application of reducing search space in the item identification process, which in turn, reduces the search time and the computational complexity in the item identification process, and processing and memory resources needed for the item identification process. Furthermore, this leads to improving the accuracy of the item identification process. For example, the user feedback may be used as additional and external information to further refine the machine learning model and increase the accuracy of the machine learning model for subsequent item identification operations. Accordingly, this represents an improvement to the efficiency, throughput, and productivity of computer systems implemented to perform the described operations.

[0029] In some embodiments, a system comprises a plurality of cameras, a memory, and a processor. Each camera is configured to capture images of at least a portion of a platform. The memory is configured to store an encoded vector library comprising a plurality of encoded vectors. Each encoded vector describes one or more attributes of a respective item. Each encoded vector is associated with a respective container category for the respective item. The processor is communicatively coupled to the plurality of cameras and the memory. The processor is configured to detect a triggering event at the platform, wherein the triggering event corresponds to a placement of an item on the platform. The processor is further configured to capture an image of the item using at least one camera from among the plurality of cameras in response to detecting the triggering event. The processor is further configured to generate a first encoded vector for the image, wherein the first encoded vector describes one or more attributes of the item. The processor is further configured to determine that the item is associated with a first container category based at least in part upon the one or more attributes of the item. The processor is further configured to identify one or more items that have been identified as having been placed inside a container associated with the first container category. The processor is further configured to display on a graphical user interface (GUI) a list of item options that comprises the one or more items. The processor is further configured to receive a selection of a first item from among the list of item options. The processor is further configured to identify the first item as being placed inside the container.

[0030] Certain embodiments of the present disclosure describe improved techniques for identifying an item placed on a platform of an imaging device. In response to detecting a placement of an item on the platform, a plurality of item identifiers are selected for the item from an encoded vector library, based on a plurality of images of the item. Each item identifier selected from the encoded vector library based on a corresponding image of the item is associated with a similarity value that is indicative of a degree of confidence that the item identifier correctly identifies the item depicted in the image. A particular item identifier is selected from the plurality of item identifiers based on the similarity values associated with the plurality of item identifiers. For example, all item identifiers that are associated with a similarity value that is less than a threshold are discarded. Among the remaining item identifiers, two item identifiers are selected that are associated with the highest and the next highest similarity values. When the difference between the highest similarity value and the next highest similarity value exceeds another threshold, the item identifier associated with the highest similarity value is assigned to the item.

[0031] The system and method described in these embodiments of the present disclosure provide a practical application of intelligently selecting a particular item identifier for an unidentified item from a plurality of item identifiers identified for the item. As described with reference to FIGS. 42 and 43, in response to detecting a triggering event corresponding to a placement of a first item on the platform of the imaging device, item tracking device captures a plurality of images of the first item, generates a plurality of cropped images of the first item based on the images, and identifies a plurality of item identifier for the first item based on the plurality of cropped images. Each item identifier that was selected based on a respective cropped image is associated with a similarity value (S) that is indicative of a degree of confidence that the item identifier correctly identifies the item depicted in the cropped image. In response to detecting that a same item identifier was not identified for a majority of the cropped images, item tracking device selects two item identifiers that are associated with the highest and the next highest similarity values. When the difference between the highest similarity value and the next highest similarity value exceeds a threshold, the item tracking device assigns the item identifier associated with the highest similarity value to the first item. This allows the item tracking device to achieve a higher accuracy in identifying an item placed on the platform, and thus, saves computing resources (e.g., processing and memory resources associated with the item tracking device) that would otherwise be used to re-identify an item that was identified incorrectly. This, for example, improves the processing efficiency associated with the processor of the item tracking device. Thus, the disclosed system and method generally improve the technology associated with automatic detection of items.

[0032] Certain embodiments of the present disclosure describe improved techniques for identifying an item placed on a platform of an imaging device. In response to detecting a triggering event corresponding to a placement of an item on a platform of an imaging device, a plurality of images of the item are captured. Each image of the item is tagged as a front image or a back image of the item. In this context, a front image of an item refers to an image of the item that includes sufficient item information to reliably identify the item. On the other hand, a back image of an item is an image of the item that includes insufficient item information to reliably identify the item. All images of the item that are tagged as back images are discarded and an item identifier is identified for the item based only on those images that are tagged as front images.

[0033] The system and method described in these embodiments of the present disclosure provide a practical application of intelligently identifying an item based on a plurality of images of the item. As described with reference to FIGS. 44, 45 and 46, in response to detecting a triggering event corresponding to a placement of a first item on the platform of the imaging device, item tracking device captures a plurality of images of the first item and generates a plurality of cropped images of the first item based on the images. Item tracking device tags each cropped image as a front image of the first item or a back image of the item. Subsequently, item tracking device discards some, but potentially all, cropped images of the first item 204A that are tagged as a back image of the first item and identifies an item identifier for the first item based primarily, if not only, on those cropped images that are tagged as front images of the item. Eliminating some or all back images of the item that do not contain unique identifiable information that can be used to reliably identify the item, before identifying the item, improves the accuracy of identification as the item is identified based primarily, if not only, on front images that include unique identifiable information of the item. This saves computing resources (e.g., processing and memory resources associated with the item tracking device) that would otherwise be used to re-identify an item that was identified incorrectly. Further, eliminating some or all back images of the item from consideration means that the item tracking device needs to process fewer images to identify the item, thus saving processing resources and time that would otherwise be used to process all cropped images of the item. This improves the processing efficiency associated with the processor of item tracking device and improves the overall user experience. Thus, the disclosed system and method generally improve the technology associated with automatic detection of items.

[0034] Certain embodiments of the present disclosure describe improved techniques for identifying an item placed on a platform of an imaging device. In response to detecting a placement of an item on a platform of an imaging device, a plurality of images of the item are captured. An encoded vector is generated for each image of the item based on attributes of the item depicted in the image. An encoded vector library lists a plurality of encoded vectors of known items. Each encoded vector from the library is tagged as corresponding to a front image of an item or a back image of an item. Each encoded vector generated for the item is compared to only those encoded vectors from the library that are tagged as front images of items. An item identifier is identified for each image of the item based on the comparison. A particular item identifier identified for a particular image is then selected and associated with the item.

[0035] The system and method described in these embodiments of the present disclosure provide a practical application of intelligently identifying an item based on a plurality of images of the item. As described with reference to FIGS. 47A, 47B and 48, in response to detecting a placement of a first item on the platform, item tracking device captures a plurality of images of the item, generates a plurality of cropped images of the item based on the images, and identifies an item identifier for each cropped image by comparing an encoded vector generated for the cropped image with primarily, if not only, those encoded vectors from the encoded vector library that are associated with a “Front” tag. This improves the overall accuracy of identifying items placed on the platform as the items are identified based primarily, if not only, on those encoded vectors from the encoded vector library that are associated with unique identifiable information relating to known items. This saves computing resources (e.g., processing and memory resources associated with the item tracking device) that would otherwise be used to re-identify an item that was identified incorrectly. Additionally, comparing encoded vectors generated based on images of an unidentified item with generally only a portion of the encoded vectors from the encoded vector library that are associated with a “Front” tag saves computing resources that would overwise be used to compare an encoded vector with all encoded vectors in the encoded vector library regardless of whether they represent front images or back images of items. This improves the processing efficiency associated with the processor of item tracking device and improves the overall user experience. Thus, the disclosed system and method generally improve the technology associated with automatic detection of items.

[0036] Certain embodiments of the present disclosure describe improved techniques for identifying an item placed on a platform of an imaging device. In response to detecting that an item has been placed on a platform of an imaging device, a plurality of images of the item are captured. All images of the item that do not include at least a threshold amount of image information associated with the item are discarded and the item is identified based only on the remaining images of the item that include at least a minimum amount (e.g., threshold amount) of image information related to the item.

[0037] The system and method described in these embodiments of the present disclosure provide a practical application of intelligently identifying an item based on a plurality of images of the item. As described with reference to FIGS. 49, 50A and 50B, in response to detecting a triggering event corresponding to a placement of a first item on the platform of the imaging device, item tracking device captures a plurality of images of the first item and generates a plurality of cropped images of the first item based on the images. For each cropped image of the unidentified first item, the item tracking device determines whether the cropped image includes at least a minimum threshold image information associated with the first item. Item tracking device discards at least some, but potentially all cropped images in which the unidentified first item does not occupy at least a minimum threshold area and identifies the first item based on the remaining cropped images. Thus, item tracking device identifies an item based primarily, if not only, on those cropped images that include sufficient image information to reliably identify the item. This improves the overall accuracy associated with identifying items placed on the platform. This saves computing resources (e.g., processing and memory resources associated with the item tracking device) that would otherwise be used to re-identify an item that was identified incorrectly. Additionally, discarding images of an item that does not include sufficient image information associated with the item means that the item tracking device needs to process fewer images to identify the item, thus saving processing resources and time that would otherwise be used to process all cropped images of the item. This improves the processing efficiency associated with the processor of item tracking device. Thus, the disclosed system and method generally improve the technology associated with automatic detection of items.

[0038] Certain embodiments of the present disclosure describe improved techniques for identifying an item placed on a platform of an imaging device. A second unidentified item that is placed on the platform is identified based on an association of the second item with an identified first item placed on the platform, wherein the association between the first item and the second item is based on a transaction history associated with a user who placed the first and second items on the platform. For example, the user may have placed the first item and the second item on the platform as part of one or more previous transactions. Based on the previous transactions, an association between the first item and the second item may be recorded as part of the user's transaction history. In a subsequent transaction, when the user places the first item and the second item on the platform, and the first item has been successfully identified, the second item is identified based on the recorded association with the first item.

[0039] The system and method described in these embodiments of the present disclosure provide a practical application of intelligently identifying an item based on a transaction history associated with a user. As described with reference to FIGS. 51, 52A and 52B, based on monitoring transactions performed by a user over a pre-configured time period, item tracking device identifies an association between a first item and a second item. The item tracking device 104 stores (e.g., as part of an encoded vector library) this user behavior identified over multiple transactions as an association between the item identifier (I1) associated with the first item and the item identifier (I2) associated with the second item. In a subsequent transaction conducted by the same user, when the item tracking device successfully identifies the first item associated with item identifier (I1) but is unable to identify the second item, the item tracking device identifies the second item as associated with item identifier (I2) based on the association between the item identifiers (I1) and (I2) stored as part of the transaction history of the user. This technique improves the overall accuracy associated with identifying items and saves computing resources (e.g., processing and memory resources associated with the item tracking device) that would otherwise be used to re-identify an item that was identified incorrectly. This improves the processing efficiency associated with the processor of item tracking device. Thus, the disclosed system and method generally improve the technology associated with automatic detection of items.

[0040] The disclosed system further contemplates an unconventional system and method for item identification using item height. More specifically, the disclosed system provides the practical application and technical improvements to the item identification and tracking techniques by reducing the search set and filtering the items based on the height of the item in question that is required to be identified.

[0041] In cases where there is a large number of items in the encoded vector library that are subject to evaluation to filter out items that do not have one or more attributes in common with the item in question, the operation to evaluate each item and filter out items is computationally complex and extensive. This leads to consuming a lot of processing and memory resources to evaluate each item. The disclosed system is configured to reduce the search space in the item identification process by filtering out items that do not have heights within a threshold range of the height of the item in question.

[0042] By narrowing down the search set and filtering out irrelevant items, the search time to identify the item is reduced and the amount of processing and memory resources required to identify the item is also reduced. Therefore, the disclosed system provides the practical application of search space reduction, time search reduction, and increasing the allocation of processing and memory resources that would otherwise be spent on evaluating irrelevant items in a larger search space from the encoded vector library. Furthermore, the disclosed system provides an additional practical application for improving the item identification techniques, and therefore, item tracking techniques. Accordingly, this represents an improvement to the efficiency, throughput, and productivity of computer systems implemented to perform the described operations.

[0043] In some embodiments, a system comprises a plurality of cameras, a memory, and a processor. Each camera is configured to capture images of at least a portion of a platform. The memory is configured to store an encoded vector library comprising a plurality of encoded vectors. Each encoded vector describes one or more attributes of a respective item. Each encoded vector is associated with a respective average height and a standard deviation from the respective average height associated with the respective item. The processor is communicatively coupled with the plurality of cameras and the memory. The processor is configured to detect a triggering event at the platform, wherein the triggering event corresponds to a placement of a first item on the platform. The processor is further configured to capture an image of the first item using a camera from among the plurality of cameras in response to detecting the triggering event. The processor is further configured to generate a first encoded vector for the image, wherein the first encoded vector describes one or more attributes of the first item. The processor is further configured to determine a height associated with the first item. The processor is further configured to identify one or more items in the encoded vector library that are associated with average heights within a threshold range from the determined height of the first item. The processor is further configured to compare the first encoded vector with a second encoded vector associated with a second item from among the one or more items. The processor is further configured to determine that the first encoded vector corresponds to the second encoded vector. The processor is further configured to determine that the first item corresponds to the second item in response to determining that the first encoded vector corresponds to the second encoded vector.

[0044] The disclosed system further contemplates an unconventional system and method for confirming the identity of an item based on item height. More specifically, the disclosed system provides the practical application and technical improvements to the item identification and tracking techniques by using the height of an item to increase the accuracy in the item identification and tracking techniques.

[0045] In an example scenario, assume that attributes of the item are used to narrow down the search set to a subset of items that may resemble or correspond to the item in question. However, a confidence score in identifying the identity of the item using the attributes of the item may be low or less than a desired value. For example, in case of using the flavor attribute of the item to filter items, the flavor of the item is usually indicated on a cover or container of the item. The machine learning algorithm processes an image of the item to detect the flavor information displayed on the cover or container of the item. However, the flavor information (e.g., shown in text) may be small in size on the container of the item. Therefore, it is challenging to detect the flavor information from an image. Similarly, various sizes of the item may appear the same or similar to each other in images of the item. For example, the image of the item may be cropped to show the item and remove side and background areas where the item is not shown. Because the image of the item is cropped, it may be difficult to differentiate between the size variations of the item, such as 8 ounce (oz), 16 oz, etc. Furthermore, similar to detecting the flavor information, detecting the size information of the item as indicated on the cover or container of the item may be challenging due to the small size of the size information. Therefore, in the examples of using flavor and size attributes to identify the item, the confidence score in determining the identity of the item may be low, e.g., less than a threshold.

[0046] The present disclosure provides a solution to this and other technical problems that are currently arising in the realm of item identification and tracking technology. For example, the disclosed system is configured to use the height of the item to confirm the identity of the item. For example, after the brand, flavor, and size attributes of the item are used to infer the identity of the item, the disclosed system may determine the confidence score associated with the identity of the item. If the confidence score is less than a threshold percentage, the system may use the height of the item to determine and confirm the identity of the item. Therefore, the disclosed system provides the practical application of improving the accuracy in the item identification techniques by leveraging the height of the item. This, in turn, reduces the search time and the computational complexity in item identification process, and processing and memory resource needed for the item identification process that would otherwise be spent in evaluating irrelevant items. Accordingly, this represents an improvement to the efficiency, throughput, and productivity of computer systems implemented to perform the described operations.

[0047] In some embodiments, a system comprises a plurality of cameras, a memory, and a processor. Each camera is configured to capture images of at least a portion of a platform. The memory is configured to store an encoded vector library comprising a plurality of encoded vectors. Each encoded vector describes one or more attributes of a respective item. Each encoded vector is associated with a respective average height and a standard deviation from the respective average height associated with the respective item. The standard deviation is a statistical measurement that quantifies an amount of dispersion or variation within a set of height values whose average is the average height. The processor is communicatively coupled with the plurality of cameras and the memory. The processor is configured to detect a triggering event at the platform, wherein the triggering event corresponds to a placement of a first item on the platform. The processor is further configured to capture an image of the first item using a camera from among the plurality of cameras in response to detecting the triggering event. The processor is further configured to generate a first encoded vector for the image, wherein the first encoded vector describes a plurality of attributes of the first item. The processor is further configured to identify a set of items in the encoded vector library that have at least one attribute in common with the first item. The processor is further configured to determine an identity of the first item based at least in part upon the plurality of attributes of the first item and the at least one attribute. The processor is further configured to determine a confidence score associated with the identity of the first item, wherein the confidence score indicates an accuracy of the identity of the first item. The processor is further configured to determine that the confidence score is less than a threshold percentage. In response to determining that the confidence score is less than the threshold percentage, the processor is further configured to determine, from the image, a height of the first item. The processor is further configured to identify one or more items from among the set of items that are associated with average heights within a threshold range from the determined height of the first item. The processor is further configured to compare the first encoded vector with a second encoded vector associated with a second item from among the one or more items. The processor is further configured to determine that the first encoded vector corresponds to the second encoded vector. The processor is further configured to determine that the first item corresponds to the second item in response to determining that the first encoded vector corresponds to the second encoded vector.

[0048] Certain embodiments of the present disclosure may include some, all, or none of these advantages. These advantages and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.

System Overview

[0118] FIG. 1 is a schematic diagram of an embodiment of an item tracking system 100 that is configured to employ digital image processing. The item tracking system 100 may employ digital image processing to identify items 204 that are placed on a platform 202 of an imaging device 102 and to assign the items 204 to a particular user. This process allows the user to obtain items 204 from a space without requiring the user to scan or otherwise manually identify the items 204 they would like to take. In one embodiment, the item tracking system 100 may be installed in a space (e.g., a store) so that shoppers need not engage in the conventional checkout process. Although the example of a store is used in this disclosure, this disclosure contemplates that the item tracking system 100 may be installed and used in any type of physical space (e.g., a room, an office, an outdoor stand, a mall, a supermarket, a convenience store, a pop-up store, a warehouse, a storage center, an amusement park, an airport, an office building, etc.). As an example, the space may be a store that comprises a plurality of items 204 that are available for purchase. The item tracking system 100 may be installed in the store so that shoppers need not engage in the conventional checkout process to purchase items from the store. In this example, the store may be a convenience store or a grocery store. In other examples, the store may not be a physical building, but a physical space or environment where shoppers may shop. For example, the store may be a “grab-and-go” pantry at an airport, a kiosk in an office building, an outdoor market at a park, etc. As another example, the space may be a warehouse or supply room that comprises a plurality of items 204 that are available for a user to use or borrow. In this example, the item tracking system 100 may be installed to allow users to checkout parts or supplies by themselves. In other examples, the item tracking system 100 may be employed for any other suitable application.

[0119] In one embodiment, the item tracking system 100 comprises one or more imaging devices 102 and an item tracking device 104 that are in signal communication with each other over a network 106. The network 106 allows communication between and amongst the various components of the item tracking system 100. This disclosure contemplates the network 106 being any suitable network operable to facilitate communication between the components of the item tracking system 100. The network 106 may include any interconnecting system capable of transmitting audio, video, signals, data, messages, or any combination of the preceding. The network 106 may include all or a portion of a local area network (LAN), a wide area network (WAN), an overlay network, a software-defined network (SDN), a virtual private network (VPN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., cellular networks, such as 4G or 5G), a Plain Old Telephone (POT) network, a wireless data network (e.g., WiFi, WiGig, WiMax, etc.), a Long Term Evolution (LTE) network, a Universal Mobile Telecommunications System (UMTS) network, a peer-to-peer (P2P) network, a Bluetooth network, a Near Field Communication (NFC) network, a Zigbee network, and/or any other suitable network.

Imaging Devices

[0120] The imaging device 102 is generally configured to capture images 122 and depth images 124 of items 204 that are placed on a platform 202 of the imaging device 102. In one embodiment, the imaging device 102 comprises one or more cameras 108, one or more three-dimensional (3D) sensors 110, and one or more weight sensors 112. Additional information about the hardware configuration of the imaging device 102 is described in FIGS. 2A-2C.

[0121] The cameras 108 and the 3D sensors 110 are each configured to capture images 122 and depth images 124 respectively of at least a portion of the platform 202. The cameras 108 are configured to capture images 122 (e.g., RGB images) of items 204. Examples of cameras 108 include, but are not limited to, cameras, video cameras, web cameras, and printed circuit board (PCB) cameras. The 3D sensors 110 are configured to capture depth images 124 such as depth maps or point cloud data for items 204. A depth image 124 comprises a plurality of pixels. Each pixel in the depth image 124 comprises depth information identifying a distance between the 3D sensor 110 and a surface in the depth image 124. Examples of 3D sensors 110 include, but are not limited to, depth-sensing cameras, time-of-flight sensors, LiDARs, structured light cameras, or any other suitable type of depth sensing device. In some embodiments, a camera 108 and a 3D sensor 110 may be integrated within a single device. In other embodiments, a camera 108 and a 3D sensor 110 may be distinct devices.

[0122] The weight sensors 112 are configured to measure the weight of items 204 that are placed on the platform 202 of the imaging device 102. For example, a weight sensor 112 may comprise a transducer that converts an input mechanical force (e.g., weight, tension, compression, pressure, or torque) into an output electrical signal (e.g., current or voltage). As the input force increases, the output electrical signal may increase proportionally. The item tracking device 104 is configured to analyze the output electrical signal to determine an overall weight for the items 204 on the weight sensor 112. Examples of weight sensors 112 include, but are not limited to, a piezoelectric load cell or a pressure sensor. For example, a weight sensor 112 may comprise one or more load cells that are configured to communicate electrical signals that indicate a weight experienced by the load cells. For instance, the load cells may produce an electrical current that varies depending on the weight or force experienced by the load cells. The load cells are configured to communicate the produced electrical signals to item tracking device 104 for processing.

Item Tracking Device

[0123] Examples of the item tracking device 104 include, but are not limited to, a server, a computer, a laptop, a tablet, or any other suitable type of device. In FIG. 1, the imaging device 102 and the item tracking device 104 are shown as two devices. In some embodiments, the imaging device 102 and the item tracking device 104 may be integrated within a single device. In one embodiment, the item tracking device 104 comprises an item tracking engine 114 and a memory 116. Additional details about the hardware configuration of the item tracking device 104 are described in FIG. 6. The memory 116 is configured to store item information 118, user account information 120, a machine learning model 126, an encoded vector library 128, and/or any other suitable type of data.

[0124] In one embodiment, the item tracking engine 114 is generally configured to process images 122 and depth images 124 to identify items 204 that are placed on the platform 202 of the imaging device 102 and to associate the identified items 204 with a user. An example of the item tracking engine 114 in operation is described in more detail below in FIGS. 3 and 7-26.

[0125] The item information 118 generally comprises information that is associated with a plurality of items. Examples of item information 118 include, but are not limited to, prices, weights, barcodes, item identifiers, item numbers, features of items, or any other suitable information that is associated with an item 204. Examples of features of an item include, but are not limited to, text, logos, branding, colors, barcodes, patterns, a shape, or any other suitable type of attributes of an item 204. The user account information 120 comprises information for one or more accounts that are associated with a user. Examples of accounts include, but are not limited to, a customer account, an employee account, a school account, a business account, a financial account, a digital cart, or any other suitable type of account. The user account information 120 may be configured to associate user information with accounts that are associated with a user. Examples of user information include, but are not limited to, a name, a phone number, an email address, an identification number, an employee number, an alphanumeric code, reward membership information, or any other suitable type of information that is associated with the user. In some embodiments, the item information 118 and/or the user account information 120 may be stored in a device (e.g. a cloud server) that is external from the item tracking device 104.

[0126] Examples of machine learning models 126 include, but are not limited to, a multi-layer perceptron, a recurrent neural network (RNN), an RNN long short-term memory (LSTM), a convolution neural network (CNN), a transformer, or any other suitable type of neural network model. In one embodiment, the machine learning model 126 is generally configured to receive an image 122 as an input and to output an item identifier based on the provided image 122. The machine learning model 126 is trained using supervised learning training data that comprises different images 122 of items 204 with their corresponding labels (e.g., item identifiers). During the training process, the machine learning model 126 determines weights and bias values that allow the machine learning model 126 to map images 122 of items 204 to different item identifiers. Through this process, the machine learning model 126 is able to identify items 204 within an image 122. The item tracking engine 114 may be configured to train the machine learning models 126 using any suitable technique as would be appreciated by one of ordinary skill in the art. In some embodiments, the machine learning model 126 may be stored and/or trained by a device that is external from the item tracking device 104.

[0127] The encoded vector library 128 generally comprises information for items 204 that can be identified by the item tracking device 104. An example of an encoded vector library 128 is shown in FIG. 16. In one embodiment, the encoded vector library 128 comprises a plurality of entries 1602. Each entry 1602 corresponds with a different item 204 that can be identified by the item tracking device 104. Referring to FIG. 16 as an example, each entry 1602 may comprise an encoded vector 1606 that is linked with an item identifier 1604 and a plurality of feature descriptors 1608. An encoded vector 1606 comprises an array of numerical values. Each numerical value corresponds with and describes a physical attribute (e.g., item type, size, shape, color, etc.) of an item 204. An encoded vector 1606 may be any suitable length. For example, an encoded vector 1606 may have a size of 1×256, 1×512, 1×1024, or any other suitable length. The item identifier 1604 uniquely identifies an item 204. Examples of item identifiers 1604 include, but are not limited to, a product name, a stock-keeping unit (SKU) number, an alphanumeric code, a graphical code (e.g., a barcode), or any other suitable type of identifier. Each of the feature descriptors 1608 describes a physical characteristic of an item 204. Examples of feature descriptors 1608 include, but are not limited to, an item type 1610, a dominant color 1612, dimensions 1614, weight 1616, or any other suitable type of descriptor that describes the physical attributes of an item 204. An item type 1610 identifies a classification for the item 204. For instance, an item type 1610 may indicate whether an item 204 is a can, a bottle, a box, a fruit, a bag, etc. A dominant color 1612 identifies one or more colors that appear on the surface (e.g., packaging) of an item 204. The dimensions 1614 may identify the length, width, and height of an item 204. In some embodiments, the dimensions 1614 may be listed in ascending order. The weight 1616 identifies the weight of an item 204. The weight 1616 may be shown in pounds, ounces, litters, or any other suitable units.

Hardware Configuration for the Imaging Device

[0128] FIG. 2A is a perspective view of an embodiment of an imaging device 102. In this example, the imaging device 102 comprises a platform 202, a frame structure 206, a plurality of cameras 108, a plurality of 3D sensors 110, and a weight sensor 112. The imaging device 102 may be configured as shown in FIG. 2A or in any other suitable configuration. In some embodiments, the imaging device 102 may further comprise additional components including, but not limited to, light, displays, and graphical user interfaces.

[0129] The platform 202 comprises a surface 208 that is configured to hold a plurality of items 204. In some embodiments, the platform 202 may be integrated with the weight sensor 112. For example, the platform 202 may be positioned on the weight sensor 112 which allows the weight sensor 112 to measure the weight of items 204 that are placed on the platform 202. As another example, the weight sensor 112 may be disposed within the platform 202 to measure the weight of items 204 that are placed on the platform 202. In some embodiments, at least a portion of the surface 208 may be transparent. In this case, a camera 108 or scanner (e.g., a barcode scanner) may be disposed below the surface 208 of the platform 202 and configured to capture images 122 or scan the bottoms of items 204 placed on the platform 202. For instance, a camera 108 or scanner may be configured to identify and read product labels and/or barcodes (e.g., SKUs) of items 204 through the transparent surface 208 of the platform 202. The platform 202 may be formed of aluminum, metal, wood, plastic, glass, or any other suitable material.

[0130] The frame structure 206 is generally configured to support and position cameras 108 and 3D sensors 110. In FIG. 2A, the frame structure 206 is configured to position a first camera 108A and a second camera 108C on the sides of the imaging device 102 with a perspective view of the items 204 on the platform 202. The frame structure 206 is further configured to position a third camera 108D on the back side of the imaging device 102 with a perspective view of the items 204 on the platform 202. In some embodiments, the frame structure 206 may further comprise a fourth camera 108 (not shown) on the front side of the imaging device 102 with a perspective view of items 204 on the platform 202. The frame structure 206 may be configured to use any number and combination of the side cameras 108A and 108C, the back side camera 108D, and the front side camera 108. For example, one or more of the identified cameras 108 may be optional and omitted. A perspective image 122 or depth image 124 is configured to capture the side-facing surfaces of items 204 placed on the platform 202. The frame structure 206 is further configured to position a third camera 108B and a 3D sensor 110 with a top view or overhead view of the items 204 on the platform 202. An overhead image 122 or depth image 124 is configured to capture upward-facing surfaces of items 204 placed on the platform 202. In other examples, the frame structure 206 may be configured to support and position any other suitable number and combination of cameras 108 and 3D sensors 110. The frame structure 206 may be formed of aluminum, metal, wood, plastic, or any other suitable material.

[0131] FIG. 2B is a perspective view of another embodiment of an imaging device 102 with an enclosure 210. In this configuration, the enclosure 210 is configured to at least partially encapsulate the frame structure 206, the cameras 108, the 3D sensors 110, and the platform 202 of the imaging device 102. The frame structure 206, the cameras 108, the 3D sensors 110, and the platform 202 may be configured similar to as described in FIG. 2A. In one embodiment, the frame structure 206 may further comprise tracks or rails 212 that are configured to allow the cameras 108 and the 3D sensors 110 to be repositionable within the enclosure 210. For example, the cameras 108A, 108C, and 108D may be repositionable along a vertical axis with respect to the platform 202 using the rails 212. Similarly, camera 108B and 3D sensor 110 may be repositionable along a horizontal axis with respect to the platform 202 using the rails 212.

[0132] FIG. 2C is a perspective view of another embodiment of an imaging device 102 with an open enclosure 214. In this configuration, the enclosure 214 is configured to at least partially cover the frame structure 206, the cameras 108, the 3D sensors 110, and the platform 202 of the imaging device 102. The frame structure 206, the cameras 108, the 3D sensors 110, and the platform 202 may be configured similar to as described in FIG. 2A. In one embodiment, the frame structure 206 may be integrated within the enclosure 214. For example, the enclosure 214 may comprise openings 216 that are configured to house the cameras 108 and the 3D sensors 110. In FIG. 2C, the enclosure 214 has a rectangular cross section with rounded edges. In other embodiments, the enclosure 214 may be configured with any other suitable shape cross section.

An Item Tracking Process

[0133] FIG. 3 is a flowchart of an embodiment of an item tracking process 300 for the item tracking system 100. The item tracking system 100 may employ process 300 to identify items 204 that are placed on the platform 202 of an imaging device 102 and to assign the items 204 to a particular user. As an example, the item tracking system 100 may employ process 300 within a store to add items 204 to a user's digital cart for purchase. As another example, the item tracking system 100 may employ process 300 within a warehouse or supply room to check out items to a user. In other examples, the item tracking system 100 may employ process 300 in any other suitable type of application where items 204 are assigned or associated with a particular user. This process allows the user to obtain items 204 from a space without having the user scan or otherwise identify the items 204 they would like to take.

[0134] At operation 302, the item tracking device 104 performs auto-exclusion for the imaging device 102. During an initial calibration period, the platform 202 may not have any items 204 placed on the platform 202. During this period of time, the item tracking device 104 may use one or more cameras 108 and 3D sensors 110 to capture reference images 122 and reference depth images 124 of the platform without any items 204 placed on the platform 202. The item tracking device 104 can then use the captured images 122 and depth images 124 as reference images to detect when an item is placed on the platform 202. For example, the item tracking device 104 may use a 3D sensor 110 that is configured with a top view or overhead view of the platform 202 to capture a reference depth image 124 of the platform 202 when no items 204 are placed on the platform 202. In this example, the captured depth image 124 may comprise a substantially constant depth value throughout the depth image 124 that corresponds with the surface 208 of the platform 202. At a later time, the item tracking device 104 can detect that an item 204 has been placed on the surface 208 of the platform 202 based on differences in depth values between subsequent depth images 124 and the reference depth image 124. As another example, the item tracking device 104 may use a camera 108 that is configured with a top view or a perspective view of the platform 202 to capture a reference image 122 of the platform when no items 204 are placed on the platform 202. In this example, the captured image 122 comprises pixel values that correspond with a scene of the platform when no items 204 are present on the platform 202. At a later time, the item tracking device 104 can detect that an item 204 has been placed on the platform 202 based on differences in the pixel values between subsequent images 122 and the reference image 122.

[0135] At operation 304, the item tracking device 104 determines whether a triggering event has been detected. A triggering event corresponds with an event that indicates that a user is interacting with the imaging device 102. For instance, a triggering event may occur when a user approaches the imaging device 102 or places an item 204 on the imaging device 102. As an example, the item tracking device 104 may determine that a triggering event has occurred in response to detecting motion using a 3D sensor 110 or based on changes in depths images 124 captured by a 3D sensor 110. For example, the item tracking device 104 can detect that an item 204 has been placed on the surface 208 of the platform 202 based on differences in depth values between depth images 124 captured by a 3D sensor 110 and the reference depth image 124. Referring to FIG. 4 as an example, FIG. 4 shows an example of a comparison between depth images 124 from an overhead view of the platform 202 of the imaging device 102 before and after placing items 204 shown in FIG. 2A on the platform 202. Depth image 124A corresponds with a reference depth image 124 that is captured when no items 204 are placed on the platform 202. Depth image 124B corresponds with a depth image 124 that is captured after items 204 are placed on the platform 202. In this example, the colors or pixel values within the depth images 124 represent different depth values. In depth image 124A, the depth values in the depth image 124A are substantially constant which means that there are no items 204 on the platform 202. In depth image 124B, the different depth values correspond with the items 204 (i.e. items 204A, 204B, and 204C) that are placed on the platform 202. In this example, the item tracking device 104 detects a triggering event in response to detecting the presence of the items 204 on the platform 202 based on differences between depth image 124A and depth image 124B. The item tracking device 104 may also use an image 122 or depth image 124 to count the number of items 204 that are on the platform 202. In this example, the item tracking device 104 determines that there are three items 204 placed on the platform 202 based on the depth image 124B. The item tracking device 104 may use the determined number of items 204 later to confirm whether all of the items 204 have been identified. This process is discussed in more detail below in operation 312.

[0136] As another example, the item tracking device 104 may determine that a triggering event has occurred in response to detecting motion using a camera 108 or based on changes in images 122 captured by a camera 108. For example, the item tracking device 104 can detect that an item 204 has been placed on the platform 202 based on differences in the pixel values between subsequent images 122 and the reference image 122. As another example, the item tracking device 104 may determine that a triggering event has occurred in response to a weight increase on the weight sensor 112 of the imaging device 102. In this case, the increase in weight measured by the weight sensor 112 indicates that one or more items 204 have been placed on the platform 202. In other examples, the item tracking device 104 may use any other suitable type of sensor or technique for detecting when a user approaches the imaging device 102 or places an item 204 on the imaging device 102.

[0137] The item tracking device 104 remains at operation 304 in response to determining that a triggering event has not been detected. In this case, the item tracking device 104 determines that a user has not interacted with the imaging device 102 yet. The item tracking device 104 will remain at operation 304 to continue to check for triggering events until a user begins interacting with the imaging device 102. The item tracking device 104 proceeds to operation 306 in response to determining that a triggering event has been detected. In this case, the item tracking device 104 determines that a user has begun interacting with the imaging device 102. The item tracking device 104 proceeds to operation 306 to begin identifying items that are placed on the platform 202 of the imaging device 102.

[0138] At operation 306, the item tracking device 104 identifies one or more cameras 108 for capturing images 122 of the items 204 on the platform 202 of the imaging device 102. The item tracking device 104 may identify cameras 108 for capturing images 122 of the items 204 based at least in part upon the pose (e.g., location and orientation) of the items 204 on the platform 202. The pose of an item 204 corresponds with the location the item 204 and how the item 204 is positioned with respect to the platform 202. Referring to the example in FIG. 2A, a first item 204A and a second item 204C are positioned in a vertical orientation with respect to the platform 202. In the vertical orientation, the identifiable features of an item 204 are primarily in the vertical orientation. Cameras 108 with a perspective view, such as cameras 108A and 108C, may be better suited for capturing images 122 of the identifiable features of item 204 that are in a vertical orientation. For instance, the item tracking device 104 may select camera 108A to capture images 122 of item 204A since most of the identifiable features of item 204A, such as branding, text, and barcodes, are located on the sides of the item 204A and are most visible using a perspective view of the item 204. Similarly, the item tracking device 104 may then select camera 108C to capture images 122 of item 204C. In this example, a third item 204B is positioned in a horizontal orientation with respect to the platform 202. In the horizontal orientation, the identifiable features of an item 204 are primarily in the horizontal orientation. Cameras 108 with a top view or overhead view, such as camera 108B, may be better suited for capturing images 122 of the identifiable features of item 204 that are in a horizontal orientation. In this case, the item tracking device 104 may select camera 108B to capture images 122 of item 204B since most of the identifiable features of item 204B are located on the top of the item 204B and are most visible from using an overhead view of the item 204B.

[0139] In one embodiment, the item tracking device 104 may determine the pose of items 204 on the platform 202 using depth images 124. Referring to FIG. 4 as an example, the depth image 124B corresponds with an overhead depth image 124 that is captured after the items 204 shown in FIG. 2A (i.e., items 204A, 204B, and 204C) are placed on the platform 202. In this example, the item tracking device 104 may use areas in the depth image 124B that correspond with each item 204 to determine the pose of the items 204. For example, the item tracking device 104 may determine the area 402 within the depth image 124B that corresponds with item 204A. The item tracking device 104 compares the determined area 402 to a predetermined area threshold value 614. The item tracking device 104 determines that an item 204 is in a vertical orientation when the determined area 402 for the item 204 is less than or equal to the predetermined area threshold value 614. Otherwise, the item tracking device 104 determines that the item 204 is in a horizontal orientation when the determined area 402 for the item 204 is greater than the predetermined area threshold value 614. In this example, the item tracking device 104 determines that items 204A and 204C are in a vertical orientation because their areas 402 and 406, respectively, are less than or equal to the area threshold value 614. The item tracking device 104 determines that item 204B is in a horizontal orientation because its area 404 is greater than the area threshold value 614. This determination means that the item tracking device 104 will select cameras 108 (e.g., cameras 108A and 108C) with a perspective view of the platform 202 to capture images 122 of items 204A and 204C. The item tracking device 104 will select a camera 108 (e.g., camera 108B) with a top view or overhead view of the platform 202 to capture images 122 of item 204B.

[0140] In one embodiment, the item tracking device 104 may identify a camera 108 for capturing images 122 of an item 204 based at least in part on the distance between the item 204 and the camera 108. For example, the item tracking device 104 may generate homographies 608 between the cameras 108 and/or the 3D sensors 110 of the imaging device 102. By generating a homography 608 the item tracking device 104 is able to use the location of an item 204 within an image 122 to determine the physical location of the item 204 with respect to the platform 202, the cameras 108, and the 3D sensors 110. This allows the item tracking device 104 to use the physical location of the item 204 to determine distances between the item 204 and each of the cameras 108 and 3D sensors 110. A homography 608 comprises coefficients that are configured to translate between pixel locations in an image 122 or depth image 124 and (x, y) coordinates in a global plane (i.e. physical locations on the platform 202). The item tracking device 104 uses homographies 608 to correlate between a pixel location in a particular camera 108 or 3D sensor 110 with a physical location on the platform 202. In other words, the item tracking device 104 uses homographies 608 to determine where an item 204 is physically located on the platform 202 based on their pixel location within an image 122 or depth image 124 from a camera 108 or a 3D sensor 110, respectively. Since the item tracking device 104 uses multiple cameras 108 and 3D sensors 110 to monitor the platform 202, each camera 108 and 3D sensor 110 is uniquely associated with a different homography 608 based on the camera's 108 or 3D sensor's 110 physical location on the imaging device 102. This configuration allows the item tracking device 104 to determine where an item 204 is physically located on the platform 202 based on which camera 108 or 3D sensor 110 it appears in and its location within an image 122 or depth image 124 that is captured by that camera 108 or 3D sensor 110. Additional information about generating a homography 608 and using a homography 608 is disclosed in U.S. Pat. No. 11,023,741 entitled, “DRAW WIRE ENCODER BASED HOMOGRAPHY” which is hereby incorporated by reference herein as if reproduced in its entirety.

[0141] As an example, the item tracking device 104 may use an image 122 or a depth image 124 from a camera 108 or 3D sensor 110, respectively, with a top view or overhead view of the platform 202 to determine the physical location of an item on the platform 202. In this example, the item tracking device 104 may determine a pixel location for the item 204 within the image 122 or depth image 124. The item tracking device 104 may then use a homography 608 to determine the physical location for the item 204 with respect to the platform 202 based on its pixel location. After determining the physical location of the item 204 on the platform 202, the item tracking device 104 may then identify which camera 108 is physically located closest to the item 204 and select the identified camera 108. Returning to the example in FIG. 2A, the item tracking device 104 may select camera 108A to capture images 122 of item 204A since camera 108A is closer to item 204A than camera 108C. Similarly, the item tracking device 104 may select camera 108C to capture images 122 of item 204C since camera 108C is closer to item 204C than camera 108A. This process ensures that the camera 108 with the best view of an item 204 is selected to capture an image 122 of the item 204.

[0142] At operation 308, the item tracking device 104 captures images 122 of the items 204 on the platform 202 using the identified cameras 108. Here, the item tracking device 104 uses the identified cameras 108 to capture images of the items 204. Referring to FIGS. 5A, 5B, and 5C as examples, the item tracking device 104 may capture a first image 122A of the item 204A, a second image 122B of item 204B, and a third image 122C of item 204C using cameras 108A, 108B, and 108C, respectively. The item tracking device 104 may collect one or more images 122 of each item 204 for processing. By using a subset of the cameras 108 available on the imaging device 102 to capture images of the items 204, the item tracking device 104 is able to reduce the number of images 122 that will be captured and processed to identify the items 204 on the platform 202. This process reduces the search space for identifying items 204 and improves the efficiency and hardware utilization of the item tracking device 104 by allowing the item tracking device 104 to process fewer images 122 to identify the item 204 instead of processing images 122 from all of the cameras 108 on the imaging device 102, which may include multiple images 122 of the same items 204. In addition, the item tracking device 104 also selects cameras 108 that are positioned to capture features that are the most useful for identifying the items 204 based on the orientation and location of the items 204, as discussed in operation 306. Examples of features include, but are not limited to, text, logos, branding, colors, barcodes, patterns, a shape, or any other suitable type of attributes of an item 204.

[0143] Returning to FIG. 3 at operation 310, the item tracking device 104 identifies the items 204 on the platform 202 based on the captured images 122. Here, the item tracking device 104 identifies an item 204 within each image 122 based on the features of the item 204 in the image 122. As an example, the machine learning model 126 may be a CNN. In this example, the machine learning model 126 includes an input layer, an output layer, and one or more hidden layers. The hidden layers include at least one convolution layer. For example, the machine learning model 126 may include the following sequence of layers: input layer, convolution layer, pooling layer, convolution layer, pooling layer, one or more fully connected layers, output layer. Each convolution layer of machine learning model 126 uses a set of convolution kernels to extract features from the pixels that form an image 122. In certain embodiments, the convolution layers of machine learning model 126 are implemented in the frequency domain, and the convolution process is accomplished using discrete Fourier transforms. This may be desirable to reduce the computational time associated with training and using machine learning model 126 for image classification purposes. For example, by converting to the frequency domain, the fast Fourier transform algorithm (FFT) may be implemented to perform the discrete Fourier transforms associated with the convolutions. Not only does the use of the FFT algorithm alone greatly reduce computational times when implemented on a single CPU (as compared with applying convolution kernels in the spatial domain), the FFT algorithm may be parallelized using one or more graphics processing units (GPUs), thereby further reducing computational times. Converting to the frequency domain may also be desirable to help ensure that the machine learning model 126 is translation and rotation invariant (e.g., the assignment made by machine learning model 126 of an image 122 to an item identifier, based on the presence of an item 204 in the image 122, should not depend on the position and/or orientation of the item 204 within image 122).

[0144] As another example, the machine learning model 126 may be a supervised learning algorithm. Accordingly, in certain embodiments, item tracking device 104 is configured to train the machine learning model 126 to assign input images 122 to any of a set of predetermined item identifiers. The item tracking device 104 may train the machine learning model 126 in any suitable manner. For example, in certain embodiments, the item tracking device 104 trains the machine learning model 126 by providing the machine learning model 126 with training data (e.g., images 122) that includes a set of labels (e.g., item identifiers) attached to the input images 122. As another example, the machine learning model 126 may be an unsupervised learning algorithm. In such embodiments, the item tracking device 104 is configured to train machine learning model 126 by providing the machine learning model 126 with a collection of images 122 and instructing the machine learning model 126 to classify these images 122 with item identifiers identified by the item tracking device 104, based on common features extracted from the images 122. The item tracking device 104 may train the machine learning model 126 any time before inputting the captured images 122 into the machine learning model 126.

[0145] After training the machine learning model 126, the item tracking device 104 may input each of the captured images 122 into the machine learning model 126. In response to inputting an image 122 in the machine learning model 126, the item tracking device 104 receives an item identifier for an item 204 from the machine learning model 126. The item identifier corresponds with an item 204 that was identified within the image 122. Examples of item identifiers include, but are not limited to, an item name, a barcode, an item number, a serial number, or any other suitable type of identifier that uniquely identifies an item 204.

[0146] In some embodiments, the item tracking device 104 may employ one or more image processing techniques without using the machine learning model 126 to identify an item 204 within an image 122. For example, the item tracking device 104 may employ object detection and/or optical character recognition (OCR) to identify text, logos, branding, colors, barcodes, or any other features of an item 204 that can be used to identify the item 204. In this case, the item tracking device 104 may process pixels within an image 122 to identify text, colors, barcodes, patterns, or any other characteristics of an item 204. The item tracking device 104 may then compare the identified features of the item 204 to a set of features that correspond with different items 204. For instance, the item tracking device 104 may extract text (e.g., a product name) from an image 122 and may compare the text to a set of text that is associated with different items 204. As another example, the item tracking device 104 may determine a dominant color within an image 122 and may compare the dominant color to a set of colors that are associated with different items 204. As another example, the item tracking device 104 may identify a barcode within an image 122 and may compare the barcode to a set of barcodes that are associated with different items 204. As another example, the item tracking device 104 may identify logos or patterns within the image 122 and may compare the identified logos or patterns to a set of logos or patterns that are associated with different items 204. In other examples, the item tracking device 104 may identify any other suitable type or combination of features and compare the identified features to features that are associated with different items 204. After comparing the identified features from an image 122 to the set of features that are associated with different items 204, the item tracking device 104 then determines whether a match is found. The item tracking device 104 may determine that a match is found when at least a meaningful portion of the identified features match features that correspond with an item 204. In response to determining that a meaningful portion of features within an image 122 match the features of an item 204, the item tracking device 104 may output an item identifier that corresponds with the matching item 204. In other embodiments, the item tracking device 104 may employ one or more image processing techniques in conjunction with the machine learning model 126 to identify an item 204 within an image 122 using any combination of the techniques discussed above.

[0147] In some embodiments, the item tracking device 104 is configured to output a confidence score 610 that indicates a probability that an item 204 has been correctly identified. For example, the item tracking device 104 may obtain a confidence score 610 from the machine learning model 126 with the determined item identifier. In this example, the machine learning model 126 outputs a confidence score 610 that is proportional to the number of features that were used or matched when determining the item identifier. As another example, the item tracking device 104 may determine a confidence score 610 based on how well identified features match the features of the identified item 204. For instance, the item tracking device 104 may obtain a confidence score 610 of 50% when half of the text identified within an image 122 matches the text associated with identified item 204. As another example, the item tracking device 104 may determine obtain a confidence score 610 of 100% when a barcode within an image 122 matches a barcode of the identified item 204. As another example, the item tracking device 104 may obtain a confidence score 610 of 25% when the dominant color within an image 122 matches a dominant color of the identified item 204. In other examples, the item tracking device 104 may obtain a confidence score 610 that is based on how well any other suitable type or combination of features matches the features of the identified item 204. Other information that can impact a confidence score 610 include, but are not limited to, the orientation of the object, the number of items on the platform 202 (e.g., a fewer number of items on the platform 202 are easier to identify than a greater number of items on the platform 202); the relative distance between items on the platform (e.g., spaced apart items on the platform 202 are easier to identify than crowded items on the platform 202); and the like. The item tracking device 104 may compare the confidence score 610 for an identified item 204 to a confidence score threshold value 612 to determine whether the item 204 has been identified. The item tracking device 104 may determine that an item 204 has not been identified when the confidence score 610 for the item 204 is less than the confidence score threshold value 612. The item tracking device 104 determines that the item 204 has been identified when the confidence score 610 for the item 204 is greater than or equal to the confidence score threshold value 612. The confidence score threshold value 612 may be set to 90%, 80%, 75%, or any other suitable value.

[0148] At operation 312, the item tracking device 104 determines whether all of the items 204 on the platform 202 have been identified. For example, the item tracking device 104 may compare the number of identified items 204 from the captured images 122 to the number of items 204 on the platform 202 that was determined in operation 304. The item tracking device 104 determines that all of the items 204 on the platform 202 have been identified when the number of items 204 identified items 204 from the captured images 122 matches the determined number of items 204 on the platform 202. Otherwise, the item tracking device 104 determines that at least one of the items 204 has not been identified when the number of items 204 identified items 204 from the captured images 122 does not match the determined number of items 204 on the platform 202.

[0149] The item tracking device 104 proceeds to operation 314 in response to determining that one or more of the items 204 on the platform 202 have not been identified. In this case, the item tracking device 104 may output a request for the user to reposition one or more items 204 on the platform 202 to assist the item tracking device 104 with identifying some of the items 204 on the platform. At operation 314, the item tracking device 104 outputs a prompt to rearrange one or more items 204 on the platform 202. As an example, one or more items 204 may be obscuring the view of an item 204 for one of the cameras 108. In this example, the item tracking device 104 may output a message on a graphical user interface that is located at the imaging device 102 with instructions for the user to rearrange the position of the items 204 on the platform 202. In some embodiments, the item tracking device 104 may also identify the locations of the one or more items 204 on the platform 202 that were not identified. For example, the item tracking device 104 may activate a light source above or below the platform 202 that illuminates an item 204 that was not recognized. In one embodiment, after outputting the message to rearrange the items 204 on the platform 202, the item tracking device 104 returns to operation 306 to restart the process of identifying the items 204 on the platform 202. This process prevents the item tracking device 104 from double counting items 204 after the items 204 have been rearranged on the platform 202.

[0150] Returning to operation 312, the item tracking device 104 proceeds to operation 316 in response to determining that all of the items 204 on the platform 202 have been identified. In some embodiments, the item tracking device 104 may validate the accuracy of detecting the identified items 204 based on the weight of the items 204 on the platform 202. For example, the item tracking device 104 may determine a first weight that is associated with the weight of the identified items 204 based on item information 118 that is associated with the identified items 204. For instance, the item tracking device 104 may use item identifiers for the identified items 204 to determine a weight that corresponds with each of the identified items 204. The item tracking device 104 may sum the individual weights for the identified items 204 to determine the first weight. The item tracking device 104 may also receive a second weight for the items 204 on the platform 202 from the weight sensor 112. The item tracking device 104 then determines a weight difference between the first weight and the second weight and compares the weight difference to a weight difference threshold value. The weight difference threshold value corresponds with a maximum weight difference between the first weight and the second weight. When the weight difference exceeds the weight difference threshold value, the item tracking device 104 may determine that there is a mismatch between the weight of the items 204 on the platform 202 of the imaging device 102 and the expected weight of the identified items 204. In this case, the item tracking device 104 may output an error message and/or return to operation 306 to restart the item tracking process. When the weight difference is less than or equal to the weight difference threshold value, the item tracking device 104 may determine that there is a match between the weight of the items 204 on the platform 202 of the imaging device 102 and the expected weight of the identified items 204. In this case, the item tracking device 104 may proceed to operation 316.

[0151] At operation 316, the item tracking device 104 checks whether any prohibited or restricted item 204 are present on the platform 202. A prohibited or restricted item 204 is an item 204 that the user is not authorized to obtain due to permission restrictions, age restrictions, or any other type of restrictions. The item tracking device 104 may compare item identifiers for the identified items 204 to a list of item identifiers for restricted or prohibited items 616. In response to determining that an item 204 matches one of the items on the list of restricted or prohibited items 616, the item tracking device 104 proceeds to operation 318 to output an alert or notification that indicates that the user is prohibited from obtaining one of the items 204 that is on the platform 202. For example, the item tracking device 104 may output an alert message that identifies the prohibited item 204 and asks the user to remove the prohibited item 204 from the platform 202 using a graphical user interface that is located at the imaging device 102. As another example, the item tracking device 104 may output an alert message that identifies the prohibited item 204 to another user (e.g. an employee) that is associated with the space. In other examples, the item tracking device 104 may output any other suitable type of alert message in response to detecting a prohibited item 204 on the platform 202.

[0152] At operation 320, the item tracking device 104 determines whether the prohibited item 204 has been removed from the platform 202. For example, the item tracking device 104 may use the weight sensors 112 to determine whether the measured weight of the item 204 on the platform 202 has decreased by an amount that corresponds with the weight of the prohibited item 204. As another example, the item tracking device 104 may use the cameras 108 and/or 3D sensors 110 to determine whether the prohibited item 204 is still present on the platform 202. In response to determining that the prohibited item 204 is still present on the platform 202, the item tracking device 104 may pause process 300 and remain at operation 320 until the prohibited item 204 has been removed from the platform 202. This process prevents the user from obtaining the prohibited item 204. The item tracking device 104 may proceed to operation 322 after the prohibited item 204 has been removed from the platform 202.

[0153] Otherwise, the item tracking device 104 proceeds to operation 322 in response to determining that no prohibited items 204 are present on the platform 202. At operation 322, the item tracking device 104 associates the items 204 with the user. In one embodiment, the item tracking device 104 may identify the user that is associated with the items 204 on the platform 202. For example, the user may identify themselves using a scanner or card reader that is located at the imaging device 102. Examples of a scanner include, but are not limited to, a QR code scanner, a barcode scanner, a near-field communication (NFC) scanner, or any other suitable type of scanner that can receive an electronic code embedded with information that uniquely identifies a person. In other examples, the user may identify themselves by providing user information on a graphical user interface that is located at the imaging device 102. Examples of user information include, but are not limited to, a name, a phone number, an email address, an identification number, an employee number, an alphanumeric code, or any other suitable type of information that is associated with the user.

[0154] The item tracking device 104 uses the information provided by the user to identify an account that is associated with the user and then to add the identified items 204 to the user's account. For example, the item tracking device 104 may use the information provided by the user to identify an account within the user account information 120 that is associated with the user. As an example, the item tracking device 104 may identify a digital cart that is associated with the user. In this example, the digital cart comprises information about items 204 that the user has placed on the platform 202 to purchase. The item tracking device 104 may add the items 204 to the user's digital cart by adding the item identifiers for the identified items 204 to the digital cart. The item tracking device 104 may also add other information to the digital cart that is related to the items 204. For example, the item tracking device 104 may use the item identifiers to look up pricing information for the identified items 204 from the stored item information 118. The item tracking device 104 may then add pricing information that corresponds with each of the identified items 204 to the user's digital cart.

[0155] After the item tracking device 104 adds the items 204 to the user's digital cart, the item tracking device 104 may trigger or initiate a transaction for the items 204. In one embodiment, the item tracking device 104 may use previously stored information (e.g., payment card information) to complete the transaction for the items 204. In this case, the user may be automatically charged for the items 204 in their digital cart when they leave the space. In other embodiments, the item tracking device 104 may collect information from the user using a scanner or card reader that is located at the imaging device 102 to complete the transaction for the items 204. This process allows the items 204 to be automatically added to the user's account (e.g., digital cart) without having the user scan or otherwise identify the items 204 they would like to take. After adding the items 204 to the user's account, the item tracking device 104 may output a notification or summary to the user with information about the items 204 that were added to the user's account. For example, the item tracking device 104 may output a summary on a graphical user interface that is located at the imaging device 102. As another example, the item tracking device 104 may output a summary by sending the summary to an email address or a user device that is associated with the user.

Hardware Configuration for the Item Tracking Device

[0156] FIG. 6 is an embodiment of an item tracking device 104 for the item tracking system 100. In one embodiment, the item tracking device 104 may comprise a processor 602, a memory 116, and a network interface 604. The item tracking device 104 may be configured as shown or in any other suitable configuration.

Processor

[0157] The processor 602 comprises one or more processors operably coupled to the memory 116. The processor 602 is any electronic circuitry including, but not limited to, state machines, one or more central processing unit (CPU) chips, logic units, cores (e.g., a multi-core processor), field-programmable gate array (FPGAs), application-specific integrated circuits (ASICs), or digital signal processors (DSPs). The processor 602 may be a programmable logic device, a microcontroller, a microprocessor, or any suitable combination of the preceding. The processor 602 is communicatively coupled to and in signal communication with the memory 116 and the network interface 604. The one or more processors are configured to process data and may be implemented in hardware or software. For example, the processor 602 may be 8-bit, 16-bit, 32-bit, 64-bit, or of any other suitable architecture. The processor 602 may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components.

[0158] The one or more processors are configured to implement various instructions. For example, the one or more processors are configured to execute item tracking instructions 606 that cause the processor to implement the item tracking engine 114. In this way, processor 602 may be a special-purpose computer designed to implement the functions disclosed herein. In an embodiment, the item tracking engine 114 is implemented using logic units, FPGAs, ASICs, DSPs, or any other suitable hardware. The item tracking engine 114 is configured to operate as described in FIGS. 1 and 3. For example, the item tracking engine 114 may be configured to perform the operations s of process 300 as described in FIG. 3.

Memory

[0159] The memory 116 is operable to store any of the information described above with respect to FIGS. 1 and 3 along with any other data, instructions, logic, rules, or code operable to implement the function(s) described herein when executed by the processor 602. The memory 116 may comprise one or more non-transitory computer-readable mediums such as computer disks, tape drives, or solid-state drives, and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory 116 may be volatile or non-volatile and may comprise a read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), dynamic random-access memory (DRAM), and static random-access memory (SRAM).

[0160] The memory 116 is operable to store item tracking instructions 606, item information 118, user account information 120, machine learning models 126, images 122, depth images 124, homographies 608, confidence scores 610, confidence score threshold values 612, area threshold values 614, a list of restricted or prohibited items 616, encoded vector libraries 128, and/or any other data or instructions. The item tracking instructions 606 may comprise any suitable set of instructions, logic, rules, or code operable to execute the item tracking engine 114. The item information 118, the user account information 120, the machine learning models 126, images 122, depth images 124, homographies 608, confidence scores 610, confidence score threshold values 612, area threshold values 614, the list of restricted or prohibited items 616, and encoded vector libraries 128 are configured similar to the item information 118, the user account information 120, the machine learning models 126, images 122, depth images 124, homographies 608, confidence scores 610, confidence score threshold values 612, area threshold values 614, the list of restricted or prohibited items 616, and encoded vector libraries 128 described in FIGS. 1-26, respectively.

Network Interface

[0161] The network interface 604 is configured to enable wired and/or wireless communications. The network interface 604 is configured to communicate data between the imaging device 102 and other devices, systems, or domains. For example, the network interface 604 may comprise an NFC interface, a Bluetooth interface, a Zigbee interface, a Z-wave interface, a radio-frequency identification (RFID) interface, a WIFI interface, a LAN interface, a WAN interface, a PAN interface, a modem, a switch, or a router. The processor 602 is configured to send and receive data using the network interface 604. The network interface 604 may be configured to use any suitable type of communication protocol as would be appreciated by one of ordinary skill in the art.

Hand Detection Process for Triggering Item Identification

[0162] FIG. 7 is a flowchart of an embodiment of a hand detection process 700 for triggering an item identification process for the item tracking system 100. The item tracking system 100 may employ process 700 to detect a triggering event that corresponds with when a user puts their hand above the platform 202 to place an item 204 on the platform 202. This process allows the item tracking device 104 to detect the presence of a user interacting with the platform 202 which can be used to initiate an item detection process such as processes 300 and 2300 described in FIGS. 3 and 23, respectively.

[0163] At operation 702, the item tracking device 104 captures a first overhead depth image 124 using a 3D sensor 110 at a first time instance. Here, the item tracking device 104 first captures an overhead depth image 124 of the platform 202 to ensure that there are no items 204 placed on the platform 202 and that there are no hands present above the platform 202 before periodically checking for the presence of a user's hand above the platform 202. The overhead depth image 124 captures any upward-facing surfaces of objects and the platform 202. Referring to FIG. 8A as an example, the item tracking device 104 may employ a 3D sensor 110 that is positioned above the platform 202 to capture an overhead depth image 124 of the platform 202. Within the overhead depth images 124 of the platform 202, the item tracking device 104 defines a region-of-interest 802 for the platform 202. The region-of-interest 802 (outlined with bold lines in FIGS. 8A-8C) identifies a predetermined range of pixels in an overhead depth image 124 that corresponds with the surface of the platform 202. The item tracking device 104 uses the defined region-of-interest 802 to determine whether any item 204 has been placed on the platform 202 or whether a user has their hand positioned above the platform 202. The region-of interest 802 is the same predetermined range of pixels for all of the depth images 124 captured by the 3D sensor 110.

[0164] Returning to FIG. 7 at operation 704, the item tracking device 104 captures a second overhead depth image 124 using the same 3D sensor 110 at a second time instance. After capturing the first overhead depth image 124, the item tracking device 104 begins periodically capturing additional overhead depth images 124 of the platform 202 to check whether a user's hand has entered the region-of-interest 802 for the platform 202. The item tracking device 104 may capture additional overhead depth images 124 every second, every ten seconds, every thirty seconds, or at any other suitable time interval. In some embodiments, the item tracking device 104 may capture the second overhead depth image 124 in response to detecting motion near the platform 202. For example, the item tracking device 104 may employ a proximity sensor that is configured to detect motion near the platform 202 before capturing the second overhead depth image 124. As another example, the item tracking device 104 may periodically capture additional overhead depth image 124 to detect motion. In this example, the item tracking device 104 compares the first overhead depth image 124 to subsequently captured overhead depth images 124 and detects motion based on differences, for example, the presence of an object, between the overhead depth images 124.

[0165] At operation 706, the item tracking device 104 determines whether an object is present within the region-of-interest 802 in the second overhead depth image 124. In one embodiment, the item tracking device 104 determines an object is present within the region-of-interest 802 based on differences between the first overhead depth image 124 and the second overhead depth image 124. Referring to FIG. 8B as an example, the item tracking device 104 compares the second overhead depth image 124 (shown in FIG. 8B) to the first overhead depth image 124 (shown in FIG. 8A) to identify differences between the first overhead depth image 124 and the second overhead depth image 124. In this example, the item tracking device 104 detects an object 804 within the in region-of-interest 802 in the second overhead depth image 124 that corresponds with the hand of a user. FIG. 8C shows a corresponding image 122 of the object 804 that is present in the second overhead depth image 124.

[0166] Returning to FIG. 7 at operation 706, the item tracking device 104 returns to operation 704 in response to determining that there is not an object present within the region-of-interest 802 in the second overhead depth image 124. In this case, the item tracking device 104 returns to operation 704 to continue periodically capturing overhead depth image 124 of the platform 202 to check where a user's hand has entered the region-of-interest 802 of the platform 202. The item tracking device 104 proceeds to operation 708 in response to determining that an object is present within the region-of-interest 802 in the second overhead depth image 124. In this case, the item tracking device 104 proceeds to operation 708 to confirm whether the object in the second overhead depth image 124 corresponds with the hand of a user.

[0167] The item tracking device 104 is configured to distinguish between an item 204 that is placed on the platform 202 and the hand of a user. When a user's hand is above the platform 202, the user's hand will typically be within the region-of-interest 802 in the second overhead depth image 124 while the user's arm remains outside of the region-of-interest 802 in the second overhead depth image 124. The item tracking device 104 uses these characteristics to confirm that a user's hand is above the platform 202, for example, when the user places an item 204 on the platform 202.

[0168] At operation 708, the item tracking device 104 determines that a first portion 806 of a first object (e.g., a user's hand and arm) is within the region-of-interest 802 in the second overhead depth image 124. Here, the item tracking device 104 confirms that a first portion 806 of the detected object which corresponds with the user's hand is within the region-of-interest 802 in the second overhead depth image 124. Returning to the example in FIG. 8B, the user's hand (shown as portion 806 of the object 804) is at least partially within the region-of-interest 802 in the second overhead depth image 124.

[0169] Returning to FIG. 7 at operation 710, the item tracking device 104 determines that a second portion 808 of the first object (e.g., a user's wrist or arm) is outside of the region-of-interest 802 while the first portion 806 of the first object (e.g., a user's hand) is within the region-of-interest 802 in the second overhead depth image 124. Returning to the example in FIG. 8B, the user's wrist and arm (shown as portion 808 of the object 804) is at least partially outside of the region-of-interest 802 while the user's hand (shown as portion 806 of the object 804) is within the region-of-interest 802 in the second overhead depth image 124. These characteristics allow the item tracking device 104 to confirm that a user's hand has been detected in the second overhead depth image 124.

[0170] After detecting the user's hand, the item tracking device 104 begins periodically capturing additional overhead depth images 124 of the platform 202 to check whether a user's hand has exited the region-of-interest 802 for the platform 202. At operation 712, the item tracking device 104 captures a third overhead depth image 124 using the 3D sensor 110 at a third time instance. The item tracking device 104 may capture additional overhead depth images 124 every second, every ten seconds, every thirty seconds, or at any other suitable time interval. In some embodiments, the item tracking device 104 may capture the third overhead depth image 124 in response to a weight change or difference on the platform 202. For example, the item tracking device 104 may use a weight sensor 112 to determine a first weight value at the first time instance when no items 204 are placed on the platform 202. The item tracking device 104 may then use the weight sensor 112 to determine a second weight value at a later time after the user places an item 204 on the platform 202. In this example, the item tracking device 104 detects a weight difference between the first weight value and the second weight value and then captures the third overhead depth image 124 in response to detecting the weight difference.

[0171] At operation 714, the item tracking device 104 determines whether the first object (i.e., the user's hand) is still present within the region-of-interest 802 in the third overhead depth image 124. Here, the item tracking device 104 may determine whether the first object is present still within the region-of-interest 802 based on differences between the second overhead depth image 124 and the third overhead depth image 124. Referring to the example in FIG. 8D, the item tracking device 104 compares the third overhead depth image 124 (shown in FIG. 8D) to the second overhead depth image 124 (shown in FIG. 8B) to identify differences between the third overhead depth image 124 and the second overhead depth image 124. In this example, the item tracking device 104 detects the first object 804 corresponding with the user's hand is no longer present within the in region-of-interest 802 in the third overhead depth image 124.

[0172] Returning to FIG. 7 at operation 714, the item tracking device 104 returns to operation 712 in response to determining that the first object 804 is still present within the region-of-interest 802 in the third overhead depth image 124. In this case, the item tracking device 104 returns to operation 712 to continue periodically checking for when the user's hand exits the region-of-interest 802 for the platform 202. The item tracking device 104 proceeds to operation 716 in response to determining that the first object 804 is no longer present within the region-of-interest 802 in the third overhead depth image 124. In this case, the item tracking device 104 begins checking for any items 204 that the user placed onto the platform 202.

[0173] At operation 716, the item tracking device 104 determines whether an item 204 is within the region-of-interest 802 in the third overhead depth image 124. When an item 204 is placed on the platform 202, the item 204 will typically be completely within the region-of-interest 802 in the third overhead depth image 124. The item tracking device 104 uses this characteristic to distinguish between an item 204 that is placed on the platform 202 and the hand of a user. Returning to the example in FIG. 8D, the item tracking device 104 detects that there is an item 204 within the region-of-interest 802 in the third overhead depth image 124.

[0174] Returning to FIG. 7 at operation 716, the item tracking device 104 returns to operation 704 in response to determining that an item 204 is not present within the region-of-interest 802 in the third overhead depth image 124. In this case, the item tracking device 104 determines that the user did not place any items 204 onto the platform 202. The item tracking device 104 returns to operation 704 to repeat the hand detection process to detect when the user's hand reenters the region-of-interest 802 for the platform 202. The item tracking device 104 proceeds to operation 718 in response to determining that an item 204 is present within the region-of-interest 802 in the third overhead depth image 124. In this case, the item tracking device 104 proceeds to operation 718 to begin capturing images 122 and/or depth images 124 of the item 204 for additional processing such as item identification.

[0175] At operation 718, the item tracking device 104 captures an image 122 of the item 204 in response to determining that the first object 804 is no longer present within the region-of-interest 802 in the third overhead depth image 124 and that an item 204 is present within the region-of-interest 802 in the third overhead depth image 124. The item tracking device 104 may use one or more cameras 108 and/or 3D sensors 110 to capture images 122 or depth images 124, respectively, of the item 204 that is placed on the platform 202.

[0176] In some embodiments, the item tracking device 104 may capture an image 122 in response to detecting a weight change or difference on the platform 202. For example, the item tracking device 104 may use a weight sensor 112 to determine a first weight value at the first time instance when no items 204 are placed on the platform 202. The item tracking device 104 may then use the weight sensor 112 to determine a second weight value at a later time after the user places the item 204 on the platform 202. In this example, the item tracking device 104 detects a weight difference between the first weight value and the second weight value and then captures image 122 in response to detecting the weight difference.

[0177] After capturing the image 122 of the item 204, the item tracking device 104 may use a process similar to processes 300 and 2300 that are described in FIGS. 3 and 23, respectively, to identify items 204 that are placed on the platform 202 based on physical attributes of the item 204 that are present in the captured image 122.

Image Cropping Process for Item Identification

[0178] FIG. 9 is a flowchart of an embodiment of an image cropping process 900 for item identification by the item tracking system 100. The item tracking system 100 may employ process 900 to isolate items 204 within an image 122. For example, when a camera 108 captures an image 122 of the platform 202, the image 122 may contain multiple items 204 that are placed on the platform 202. To improve the accuracy when identifying an item 204, the item tracking device 104 first crops the image 122 to isolate each item 204 within the image 122. Cropping the image 122 generates a new image 122 (i.e., a cropped image 122) that comprises pixels from the original image 122 that correspond with an item 204. The item tracking device 104 repeats the process to create a set of cropped images 122 that each correspond with an item 204.

[0179] At operation 902, the item tracking device 104 captures a first image 122 of an item 204 on the platform 202 using a camera 108. The item tracking device 104 may use a camera 108 with an overhead, perspective, or side profile view to capture the first image 122 of the item 204 on the platform 202. As an example, the camera 108 may be configured with an overhead view to capture upward-facing surfaces of the item 204. As another example, the camera 108 may be configured with a perspective or side profile view to capture the side-facing surfaces of the item 204.

[0180] At operation 904, the item tracking device 104 identifies a region-of-interest 1002 for the item 204 in the first image 122. The region-of-interest 1002 comprises a plurality of pixels that correspond with an item 204 in the first image 122. An example of a region-of-interest 1002 is a bounding box. In some embodiments, the item tracking device 104 may employ one or more image processing techniques to identify a region-of-interest 1002 for an item 204 within the first image 122. For example, the item tracking device 104 may employ object detection and/or OCR to identify text, logos, branding, colors, barcodes, or any other features of an item 204 that can be used to identify the item 204. In this case, the item tracking device 104 may process the pixels within the first image 122 to identify text, colors, barcodes, patterns, or any other characteristics of an item 204. The item tracking device 104 may then compare the identified features of the item 204 to a set of features that correspond with different items 204. For instance, the item tracking device 104 may extract text (e.g., a product name) from the first image 122 and may compare the text to a set of text that is associated with different items 204. As another example, the item tracking device 104 may determine a dominant color within the first image 122 and may compare the dominant color to a set of colors that are associated with different items 204. As another example, the item tracking device 104 may identify a barcode within the first image 122 and may compare the barcode to a set of barcodes that are associated with different items 204. As another example, the item tracking device 104 may identify logos or patterns within the first image 122 and may compare the identified logos or patterns to a set of logos or patterns that are associated with different items 204. In other examples, the item tracking device 104 may identify any other suitable type or combination of features and compare the identified features to features that are associated with different items 204.

[0181] After comparing the identified features from the first image 122 to the set of features that are associated with different items 204, the item tracking device 104 then determines whether a match is found. The item tracking device 104 may determine that a match is found when at least a meaningful portion of the identified features match features that correspond with an item 204. In response to determining that a meaningful portion of features within the first image 122 matches the features of an item 204, the item tracking device 104 identifies a region-of-interest 1002 that corresponds with the matching item 204. In other embodiments, the item tracking device 104 may employ any other suitable type of image processing techniques to identify a region-of-interest 1002. FIGS. 10A, 10B, 10C, and 10D illustrate examples of region-of-interest 1002 for the item 204.

[0182] At operation 906, the item tracking device 104 determines a first number of pixels in the region-of-interest 1002 that correspond with the item 204 in the first image 122. Here, the item tracking device 104 counts the number of pixels within the plurality of pixels in the identified region-of-interest 1002. The number of pixels within the region-of-interest 1002 is proportional to how much of the first item 204 was detected within the first image 122. For example, a greater number of pixels within the region-of-interest 1002 indicates that a larger portion of the item 204 was detected within the first image 122. Alternatively, a fewer number of pixels within the region-of-interest 1002 indicates that a smaller portion of the item 204 was detected within the first image 122. In some instances, a small number of pixels within the region-of-interest 1002 may indicate that only a small portion of the item 204 was visible to the selected camera 108 or that the region-of-interest 1002 was incorrectly identified. The item tracking device 104 proceeds to operation 908 to determine whether the region-of-interest 1002 was correctly identified.

[0183] At operation 908, the item tracking device 104 captures a first depth image 124 of the item 204 on the platform using a 3D sensor 110. Here, the item tracking device 104 uses a 3D sensor 110 to capture a first depth image 124 with a similar view of the item 204 that was captured by the camera 108 in operation 902. For example, the item tracking device 104 may use a 3D sensor 110 that is configured with an overhead view of the item 204 when a camera 108 with an overhead view of the item 204 is used to capture the first image 122. As another example, the item tracking device 104 may use a 3D sensor 110 that is configured with a perspective or side profile view of the item 204 when a camera 108 with a perspective or side profile view of the item 204 is used to capture the first image 122. In other examples, the item tracking device 104 may use a 3D sensor 110 that has any other type of view of the item 204 that is similar the view captured in the first image 122. FIGS. 10A, 10B, 10C, and 10D illustrate examples of the first depth image 124.

[0184] At operation 910, the item tracking device 104 determines a second number of pixels in the first depth image 124 corresponding with the item 204. Here, the item tracking device 104 counts the number of pixels within the first depth image 124 that correspond with the item 204. In some embodiments, the item tracking device 104 may use a depth threshold value to distinguish between pixels corresponding with the item 204 and other items 204 or the platform 202. For example, the item tracking device 104 may set a depth threshold value that is behind the surface of the item 204 that is facing the 3D sensor 110. After applying the depth threshold value, the remaining pixels in the first depth image 124 correspond with the item 204. The item tracking device 104 may then count the remaining number of pixels within the first depth image 124 after applying the depth threshold value to the first depth image 124.

[0185] At operation 912, the item tracking device 104 determines a difference between the first number of pixels and the second number of pixels. Here, the item tracking device 104 the difference between the number of pixels for the item 204 from the region-of-interest 1002 and the number of pixels for the item 204 from the first depth image 124 to determine how similar the two values are to each other. For example, the item tracking device 104 may subtract the first number of pixels from the second number of pixels to determine the difference between the two values. In this example, the item tracking device 104 may use the absolute value of the difference between the two values.

[0186] At operation 914, the item tracking device 104 determines whether the difference is less than or equal to a difference threshold value. The distance threshold value is a user-defined value that identifies a maximum pixel difference for the identified region-of-interest 1002 to be considered valid for additional processing. An invalid region-of-interest 1002 means that the difference between the number of pixels for the item 204 in the region-of-interest 1002 and the number of pixels for the item 204 in the first depth image 124 is too great. An invalid region-of-interest 1002 indicates that the region-of-interest 1002 captures a smaller portion of the item 204 than is visible from the camera 108 and the 3D sensor 110. Since an invalid region-of-interest 1002 only captures a small portion of the item 204, the region-of-interest 1002 may not be suitable for subsequent image processing after cropping the first image 122 using the region-of-interest 1002. Referring to FIG. 10A as an example of an invalid region-of-interest 1002, the item tracking device 104 identifies a first region-of-interest 1002A and the first depth image 124 of the item 204. In this example, the difference between the number of pixels for the item 204 in the region-of-interest 1002 and the number of pixels for the item 204 in the first depth image 124 is greater than the difference threshold value. An example of the first region-of-interest 1002A overlaid with the item 204 in the first depth image 124 is shown in FIG. 10B.

[0187] A valid region-of-interest 1002 means that the difference between the number of pixels for the item 204 in the region-of-interest 1002 and the number of pixels for the item 204 in the first depth image 124 is within a predetermined tolerance level (i.e. the difference threshold value). Referring to FIG. 10C as an example of a valid region-of-interest 1002, the item tracking device 104 identifies a second region-of-interest 1002B and the first depth image 124 of the item 204. In this example, the difference between the number of pixels for the item 204 in the region-of-interest 1002 and the number of pixels for the item 204 in the first depth image 124 is less than or equal to the difference threshold value. An example of the second region-of-interest 1002B overlaid with the item 204 in the first depth image 124 is shown in FIG. 10D.

[0188] Returning to FIG. 9, the item tracking device 104 returns to operation 904 in response to determining that the difference is greater than the difference threshold value. In this case, the item tracking device 104 discards the current region-of-interest 1002 and returns to operation 904 to obtain a new region-of-interest 1002 for the item 204. The item tracking device 104 proceeds to operation 916 in response to determining that the difference is less than or equal to the difference threshold value. In this case, the item tracking device 104 proceeds to operation 916 to crop the first image 122 using the identified region-of-interest 1002.

[0189] At operation 916, the item tracking device 104 crops the first image 122 based on the region-of-interest 1002. After determining that the region-of-interest 1002 is valid additional processing, the item tracking device 104 crops the first image 122 by extracting the pixels within the region-of-interest 1002 from the first image 122. By cropping the first image 122, the item tracking device 104 generates a second image 122 that comprises the extracted pixels within the region-of-interest 1002 of the first image 122.

[0190] At operation 918, the item tracking device 104 outputs the second image 122. After generating the second image 122, the item tracking device 104 may output the second image 122 for additional processing. For example, the item tracking device 104 may output the second image 122 by inputting or loading the second image 122 into a machine learning model 126 to identify the item 204 using a process similar to process 2300 that is described in FIG. 23. As another example, the item tracking device 104 may associate the second image 122 with feature descriptors 1608 (e.g. an item type 1610, dominant color 1612, dimensions 1614, weight 1616) for the item 204 using a process similar to process 2300 that is described in FIG. 23.

Item Location Detection Process

[0191] FIG. 11 is a flowchart of an embodiment of an item location detection process 1100 for the item tracking system 100. The item tracking system 100 may employ process 1100 to identify groups of images 122 that correspond with the same item 204. The item tracking device 104 typically uses multiple cameras 108 to capture images 122 of the items 204 on the platform 202 from multiple perspectives. This process allows the item tracking device 104 to use redundancy to ensure that all of the items 204 are visible in at least one of the captured images 122. Since each camera 108 has a different physical location and perspective of the platform 202, the items 204 will appear in different locations in each of the captured images 122. To resolve this issue, the item tracking device 104 uses homographies 608 to cluster together images 122 of the same item 204 based on each item's 204 physical location on the platform 202. This process allows the item tracking device 104 to generate a set of images 122 for each item 204 that is on the platform 202 using the captured images 122 from the multiple camera perspectives.

Generating a Homography

[0192] The item tracking device 104 is configured to generate and use homographies 608 to map pixels from the cameras 108 and 3D sensors 110 to the platform 202. An example of a homography 608 is described below in FIGS. 12A and 12B. By generating a homography 608 the item tracking device 104 is able to use the location of an item 204 within an image 122 to determine the physical location of the item 204 with respect to the platform 202, the cameras 108, and the 3D sensors 110. This allows the item tracking device 104 to use the physical location of the item 204 to cluster images 122 and depth images 124 of an item 204 together for processing. Each homography 608 comprises coefficients that are configured to translate between pixel locations in an image 122 or depth image 124 and (x,y) coordinates in a global plane (i.e. physical locations on the platform 202). Each image 122 and depth image 124 comprises a plurality of pixels. The location of each pixel within an image 122 or depth image 124 is described by its pixel location 1202 which identifies a pixel row and a pixel column for a pixel where the pixel is located within an image 122 or depth image 124.

[0193] The item tracking device 104 uses homographies 608 to correlate between a pixel location in a particular camera 108 or 3D sensor 110 with a physical location on the platform 202. In other words, the item tracking device 104 uses homographies 608 to determine where an item 204 is physically located on the platform 202 based on their pixel location 1202 within an image 122 or depth image 124 from a camera 108 or a 3D sensor 110, respectively. Since the item tracking device 104 uses multiple cameras 108 and 3D sensors 110 to monitor the platform 202, each camera 108 and 3D sensor 110 is uniquely associated with a different homography 608 based on the camera's 108 or 3D sensor's 110 physical location on the imaging device 102. This configuration allows the item tracking device 104 to determine where an item 204 is physically located on the platform 202 based on which camera 108 or 3D sensor 110 it appears in and its location within an image 122 or depth image 124 that is captured by that camera 108 or 3D sensor 110. In this configuration, the cameras 108 and the 3D sensors 110 are configured to capture images 122 and depth images 124, respectively, of at least partially overlapping portions of the platform 202.

[0194] Referring to FIG. 12A, a homography 608 comprises a plurality of coefficients configured to translate between pixel locations 1202 in an image 122 or a depth image 124 and physical locations (e.g. (x,y) coordinates 1204) in a global plane that corresponds with the top surface of the platform 202. In this example, the homography 608 is configured as a matrix and the coefficients of the homography 608 are represented as H₁₁, H₁₂, H₁₃, H₁₄, H₂₁, H₂₂, H₂₃, H₂₄, H₃₁, H₃₂, H₃₃, H₃₄, H₄₁, H₄₂, H₄₃, and H₄₄. The item tracking device 104 may generate the homography 608 by defining a relationship or function between pixel locations 1202 in an image 122 or a depth image 124 and physical locations (e.g. (x,y) coordinates 1204) in the global plane using the coefficients. For example, the item tracking device 104 may define one or more functions using the coefficients and may perform a regression (e.g. least squares regression) to solve for values for the coefficients that project pixel locations 1202 of an image 122 or a depth image 124 to (x,y) coordinates 1204 in the global plane. Each (x,y) coordinate 1204 identifies an x-value and a y-value in the global plane where an item is located on the platform 202. In other examples, the item tracking device 104 may solve for coefficients of the homography 608 using any other suitable technique. In the example shown in FIG. 5A, the z-value at the pixel location 1202 may correspond with a pixel value that represents a distance, depth, elevation, or height. In this case, the homography 608 is further configured to translate between pixel values in a depth image 124 and z-coordinates (e.g. heights or elevations) in the global plane.

[0195] The item tracking device 104 may use the inverse of the homography 608 to project from (x,y) coordinates 1204 in the global plane to pixel locations 1202 in an image 122 or depth image 124. For example, the item tracking device 104 receives an (x,y) coordinate 1204 in the global plane for an object. The item tracking device 104 identifies a homography 608 that is associated with a camera 108 or 3D sensor 110 where the object is seen. The item tracking device 104 may then apply the inverse homography 608 to the (x,y) coordinate 1204 to determine a pixel location 1202 where the object is located in the image 122 or depth image 124. The item tracking device 104 may compute the matrix inverse of the homograph 608 when the homography 608 is represented as a matrix. Referring to FIG. 12B as an example, the item tracking device 104 may perform matrix multiplication between an (x,y) coordinates 1204 in the global plane and the inverse homography 608 to determine a corresponding pixel location 1202 in the image 122 or depth image 124.

[0196] Additional information about generating a homography 608 and using a homography 608 is disclosed in U.S. Pat. No. 11,023,741 entitled, “DRAW WIRE ENCODER BASED HOMOGRAPHY” which is hereby incorporated by reference herein as if reproduced in its entirety.

Using Homographies to Determine an Item's Location

[0197] Returning to FIG. 11, after generating homographies 608 for the cameras 108 and/or 3D sensors 110, the item tracking device 104 may then use the homographies 608 to cluster images 122 and depth images 124 of items 204 together for processing. At operation 1102, the item tracking device 104 captures a first image 122 of an item 204 using a first camera 108. The first camera 108 may be configured upward-facing surfaces and/or side surfaces of the items 204 on the platform 202. Referring to FIG. 13A, the item tracking device 104 uses a first camera 108 to capture a first image 1302 of items 204A and 204B that are on the platform 202.

[0198] Returning to FIG. 11 at operation 1104, the item tracking device 104 identifies a first region-of-interest 1304 for an item 204 in the first image 122. The first region-of-interest 1304 comprises a plurality of pixels that correspond with the item 204 in the first image 122. An example of a region-of-interest 1304 is a bounding box. In some embodiments, the item tracking device 104 may employ one or more image processing techniques to identify a region-of-interest 1304 for an item 204 within the first image 122. For example, the item tracking device 104 may employ object detection and/or OCR to identify text, logos, branding, colors, barcodes, or any other features of an item 204 that can be used to identify the item 204. In this case, the item tracking device 104 may process pixels within an image 122 to identify text, colors, barcodes, patterns, or any other characteristics of an item 204. The item tracking device 104 may then compare the identified features of the item 204 to a set of features that correspond with different items 204. For instance, the item tracking device 104 may extract text (e.g. a product name) from an image 122 and may compare the text to a set of text that is associated with different items 204. As another example, the item tracking device 104 may determine a dominant color within an image 122 and may compare the dominant color to a set of colors that are associated with different items 204. As another example, the item tracking device 104 may identify a barcode within an image 122 and may compare the barcode to a set of barcodes that are associated with different items 204. As another example, the item tracking device 104 may identify logos or patterns within the image 122 and may compare the identified logos or patterns to a set of logos or patterns that are associated with different items 204. In other examples, the item tracking device 104 may identify any other suitable type or combination of features and compare the identified features to features that are associated with different items 204.

[0199] After comparing the identified features from an image 122 to the set of features that are associated with different items 204, the item tracking device 104 then determines whether a match is found. The item tracking device 104 may determine that a match is found when at least a meaningful portion of the identified features match features that correspond with an item 204. In response to determining that a meaningful portion of features within an image 122 match the features of an item 204, the item tracking device 104 may identify a region-of-interest 1304 that corresponds with the matching item 204. In other embodiments, the item tracking device 104 may employ any other suitable type of image processing techniques to identify a region-of-interest 1304. Returning to the example in FIG. 13A, the item tracking device 104 identifies a first region-of-interest 1304A corresponding with the first item 204A and a second region-of-interest 1304B corresponding with the second item 204B in the first image 1302.

[0200] Returning to FIG. 11 at operation 1106, the item tracking device 104 identifies a first pixel location 1202 within the first region-of-interest 1304. The pixel location 1202 may be any pixel within the first region-of-interest 1304. In some embodiments, the item tracking device 104 may identify a pixel location 1202 that is closest to the platform 202. For example, the item tracking device 104 may identify a pixel location 1202 at a midpoint on a lower edge of the region-of-interest 1304. Returning to the example in FIG. 13A, the item tracking device 104 may identify a pixel location 1202A within the first region-of-interest 1304A for the first item 204A and a pixel location 1202B within the second region-of-interest 1304B for the second item 204B.

[0201] Returning to FIG. 11 at operation 1108, the item tracking device 104 applies a first homography 608 to the first pixel location 1202 to determine a first (x,y) coordinate 1204 on the platform 202 for the item 204. For example, the item tracking device 104 identifies a homography 608 that is associated with the first camera 108 and then applies the identified homography 608 to the pixel location 1202 for each item 204 to determine their corresponding (x,y) coordinate 1204 on the platform 202.

[0202] At operation 1110, the item tracking device 104 captures a second image 122 of the item 204 using a second camera 108. Here, the item tracking device 104 uses a different camera 108 to capture a different view of the items 204 on the platform 202. The second camera 108 may be configured upward-facing surfaces and/or side surfaces of the items 204 on the platform 202. Referring to the example in FIG. 13B, the item tracking device 104 uses a second camera 108 to capture a second image 1306 of the items 204A and 204B that are on the platform 202. In this example, the second camera 108 is on the opposite side of the platform 202 from the first camera 108. In this example, the first camera 108 captures a first side of the items 204 on the platform 202 and the second camera 108 captures an opposing side of the items 204 on the platform 202. In other examples, the second camera 108 may be in any other suitable location.

[0203] Returning to FIG. 11 at operation 1112, the item tracking device 104 identifies a second region-of-interest 1304 for the item 204 in the second image 122. The second region-of-interest 1304 comprises a second plurality of pixels that correspond with the item 204 in the second image 122. The item tracking device 104 may repeat the process described in operation 1104 to identify the second region-of-interest 1304. Returning to the example in FIG. 13B, the item tracking device 104 identifies a third region-of-interest 1304C corresponding with the first item 204A and a fourth region-of-interest 1304D corresponding with the second item 204B in the second image 1306.

[0204] Returning to FIG. 11 at operation 1114, the item tracking device 104 identifies a second pixel location 1202 within the second region-of-interest 1304. Returning to the example in FIG. 13B, the item tracking device 104 may identify a pixel location 1202C within the third region-of-interest 1304C for the first item 204A and a pixel location 1202D within the fourth region-of-interest 1304D for the second item 204B.

[0205] Returning to FIG. 11 at operation 1116, the item tracking device 104 applies a second homography 608 to the second pixel location 1202 to determine a second (x, y) coordinate 1204 on the platform 202 for the item 204. Here, the item tracking device 104 identifies a homography 608 that is associated with the second camera 108 and then applies the identified homography 608 to the pixel location 1202 for each item 204 to determine their corresponding (x,y) coordinate 1204 on the platform 202.

[0206] The item tracking device 104 may repeat this process for any other suitable number of cameras 108. Referring to FIG. 13C as another example, the item tracking device 104 may use third camera 108 to capture a third image 1308 of the items 204 on the platform 202. The item tracking device 104 may then identify regions-of-interest 1304 and pixel locations 1202 for each item 204. In this example, the item tracking device 104 identifies a region-of-interest 1304E and a pixel location 1202E for the first item 204A and a region-of-interest 1304F and a pixel location 1202F for the second item 204B. After determining the pixel locations 1202 for the items 204, the item tracking device 104 then identifies a homography 608 that is associated with the third camera 108 and applies the identified homography 608 to the pixel location 1202 for each item 204 to determine their corresponding (x,y) coordinate 1204 on the platform 202.

[0207] Returning to FIG. 11 at operation 1118, the item tracking device 104 determines a distance 1402 between the first (x,y) coordinate 1204 and the second (x,y) coordinate 1204. Referring to FIG. 14 as an example, FIG. 14 shows an overhead view of the platform 202 with the (x,y) coordinates 1204 for each item 204 projected onto the platform 202. In this example, (x,y) coordinates 1204A, 1204B, and 1204C are associated with the first item 204A and (x,y) coordinates 1204D, 1204E, and 1204F are associated with the second item 204B. The item tracking device 104 is configured to iteratively select pairs of (x,y) coordinates 1204 and to determine a distance 1402 between a pair of (x,y) coordinates 1204. In one embodiment, the item tracking device 104 is configured to determine a Euclidian distance between a pair of (x,y) coordinates 1204.

[0208] Returning to FIG. 11 at operation 1120, the item tracking device 104 determines whether the distance 1402 is less than or equal to a distance threshold value. The distance threshold value identifies a maximum distance between a pair of (x,y) coordinates 1204 to be considered members of the same cluster 1404 for an item 204. The distance threshold value is a user-defined value that may be set to any suitable value. The distance threshold value may be in units of inches, centimeters, millimeters, or any other suitable units. The item tracking device 104 compares the distance 1402 between a pair of (x,y) coordinates 1204 and the distance threshold value and determines whether the distance 1402 between the pair of (x,y) coordinates 1204 is less than the distance threshold value.

[0209] The item tracking device 104 terminates process 1100 in response to determining that the distance 1402 is greater than the distance threshold value. In this case, the item tracking device 104 determines that the pair of (x,y) coordinates 1204 are not members of the same cluster 1404 for an item 204. In some embodiments, the item tracking device 104 may not terminate process 1100, but instead will select another pair of (x,y) coordinates 1204 when additional (x,y) coordinates 1204 are available to compare to the distance threshold value.

[0210] The item tracking device 104 proceeds to operation 1122 in response to determining that the distance 1402 is less than or equal to the distance threshold value. In this case, the item tracking device 104 determines that the pair of (x,y) coordinates 1204 are members of the same cluster 1404 for an item 204. At operation 1122, the item tracking device 104 associates the pixels within the first region-of-interest 1304 from the first image 122 and the pixels within the second region-of-interest 1304 from the second image 122 with a cluster 1404 for the item 204. Referring to FIG. 14 as an example, the item tracking device 104 may identify a first cluster 1404A for the first item 204A and a second cluster 1404B for the second item 204B. The first cluster 1404A is associated with (x,y) coordinates 1204A, 1024B, and 1204C and region-of-interest 1304A, 1304C, and 1304E. The second cluster 1404B is associated with (x,y) coordinates 1204D, 1024E, and 1204F and region-of-interest 1304B, 1304D, and 1304F.

[0211] Returning to FIG. 11 at operation 1124, the item tracking device 104 outputs the pixels within the first region-of-interest 1304 from the first image 122 and the pixels within the second region-of-interest 1304 from the second image 122. In one embodiment, the item tracking device 104 will crop the captured images 122 by extracting the pixels within identified regions-of-interest 1304 from the images 122. By cropping an image 122, the item tracking device 104 generates a new image 122 that comprises the extracted pixels within a region-of-interest 1304 of the original image 122. This process allows the item tracking device 104 to generate a new set of images 122 for an item 204 that each comprise the extracted pixels from the identified regions-of-interest 1304 that were associated with the item 204. The item tracking device 104 may output the new images 122 for the item 204 for additional processing. For example, the item tracking device 104 may output the images 122 by inputting or loading them into a machine learning model 126 to identify the item 204 based on the physical attributes of the item 204 in the images 122 using a process similar to process 2300 that is described in FIG. 23.

[0212] In some embodiments, the item tracking device 104 may also associate any identified feature descriptors with the images 122 for the item 204 and output the feature descriptors with the images 122 of the item 204. For example, while determining the region-of-interest 1304 for an item 204, the item tracking device 104 may identify an item type for the item 204. In this example, the item tracking device 104 may associate the item type with the region-of-interest 1304 and output the item type with the image 122 of the item 204 that is generated based on the region-of-interest 1304. As another example, the item tracking device 104 may obtain a weight for the item 204 using the weight sensor 112. In this example, the item tracking device 104 may associate the weight with the region-of-interest 1304 and output the weight with the image 122 of the item 204 that is generated based on the region-of-interest 1304. In other examples, the item tracking device 104 may be configured to identify and associate any other suitable type of feature descriptors with a region-of-interest 1304 before outputting the region-of-interest 1304.

Search Space Reduction Process for an Encoded Vector Library

[0213] FIG. 15 is a flowchart of an embodiment of a search space reduction process 1500 for an encoded vector library 128. The item tracking system 100 may employ process 1500 to filter the entries 1602 in the encoded vector library 128 to reduce the amount of items 204 that are considered when attempting to identify an item 204 that is placed on the platform 202. This process reduces the amount of time required to search for a corresponding entry 1602 in the encoded vector library 128 as well as improves the accuracy of the results from identifying an entry 1602 in the encoded vector library 128.

[0214] At operation 1502, the item tracking device 104 obtains feature descriptors 1608 for an item 204. Each of the feature descriptors 1608 describes the physical characteristics or attributes of an item 204. Examples of feature descriptors 1608 include, but are not limited to, an item type 1610, a dominant color 1612, dimensions 1614, weight 1616, or any other suitable type of descriptor that describes an item 204. In one embodiment, the item tracking device 104 may obtain feature descriptors using a process similar to the process described in operation 1104 of FIG. 11. For example, the item tracking device 104 may employ object detection and/or OCR to identify text, logos, branding, colors, barcodes, or any other features of an item 204 that can be used to identify the item 204. In some embodiments, the item tracking device 104 may determine the dimensions of the item 204 using a process similar to process 1800 that is described in FIG. 18. The item tracking device 104 may determine the weight of the item 204 using a weight sensor 112. In other embodiments, the item tracking device 104 may use any other suitable process for determining feature descriptors for the item 204.

[0215] At operation 1504, the item tracking device 104 determines whether the feature descriptors 1608 identify an item type 1610 for the item 204. Here, the item tracking device 104 determines whether any information associated with an item type 1610 for the item 204 is available. An item type 1610 identifies a classification for the item 204. For instance, an item type 1610 may indicate whether an item 204 is a can, a bottle, a box, a fruit, a bag, etc. The item tracking device 104 proceeds to operation 1506 in response to determining that the feature descriptors 1608 identify an item type 1610 for the item 204. In this case, the item tracking device 104 uses the item type 1610 to filter the encoded vector library 128 to reduce the number of entries 1602 in the encoded vector library 128 before attempting to identify the item 204.

[0216] At operation 1506, the item tracking device 104 filters the encoded vector library 128 based on the item type 1610. Referring to FIG. 16 as an example, the encoded vector library 128 comprises a plurality of entries 1602. Each entry 1602 corresponds with a different item 204 that can be identified by the item tracking device 104. Each entry 1602 may comprise an encoded vector 1606 that is linked with an item identifier 1604 and a plurality of feature descriptors 1608. An encoded vector 1606 comprises an array of numerical values. Each numerical value corresponds with and describes an attribute (e.g. item type, size, shape, color, etc.) of an item 204. An encoded vector 1606 may be any suitable length. For example, an encoded vector 1606 may have a size of 1×256, 1×512, 1×1024, or any other suitable length. The item identifier 1604 uniquely identifies an item 204. Examples of item identifiers 1604 include, but are not limited to, a product name, an SKU number, an alphanumeric code, a graphical code (e.g. a barcode), or any other suitable type of identifier. In this example, the item tracking device 104 uses the item type 1610 to filter out or remove any entries 1602 in the encoded vector library 128 that do not contain the same item type 1610. This process reduces the number of entries 1602 in the encoded vector library 128 that will be considered when identifying the item 204.

[0217] Returning to FIG. 15 at operation 1504, the item tracking device 104 proceeds to operation 1508 in response to determining that the feature descriptors 1608 do not identify an item type 1610. In this case, the item tracking device 104 checks for other types of feature descriptors 1608 that can be used to filter the entries 1602 in the encoded vector library 128. At operation 1508, the item tracking device 104 determines whether the feature descriptors 1608 identify a dominant color 1612 for the item 204. A dominant color 1612 identifies one or more colors that appear on the surface (e.g. packaging) of an item 204.

[0218] The item tracking device 104 proceeds to operation 1510 in response to determining that the feature descriptors 1608 identify a dominant color 1612 for the item 204. In this case, the item tracking device 104 proceeds to operation 1510 to reduce the number of entries 1602 in the encoded vector library 128 based on the dominant color 1612 of the item 204. At operation 1510, the item tracking device 104 filters the encoded vector library 128 based on the dominant color 1612 of the item 204. Here, the item tracking device 104 uses the dominant color 1612 to filter out or remove any entries 1602 in the encoded vector library 128 that do not contain the same dominant color 1612.

[0219] Returning to operation 1508, the item tracking device 104 proceeds to operation 1512 in response to determining that the feature descriptors 1608 do not identify a dominant color 1612 for the item 204. At operation 1512, the item tracking device 104 determines whether the feature descriptors 1608 identify dimensions 1614 for the item 204. The dimensions 1614 may identify the length, width, and height of an item 204. In some embodiments, the dimensions 1614 may be listed in ascending order.

[0220] The item tracking device 104 proceeds to operation 1514 in response to determining that the feature descriptors 1608 identify dimensions 1614 for the item 204. In this case, the item tracking device 104 proceeds to operation 1514 to reduce the number of entries 1602 in the encoded vector library 128 based on the dimensions 1614 of the item 204. At operation 1514, the item tracking device 104 filters the encoded vector library 128 based on the dimensions 1614 of the item 204. Here, the item tracking device 104 uses the dimensions 1614 to filter out or remove any entries 1602 in the encoded vector library 128 that do not contain the same dimensions 1614 as the item 204 or within a predetermined tolerance of the dimensions 1614 of the item 204. In some embodiments, this dimensions 1614 of the item 204 may be listed in ascending order to make the comparison easier between the dimensions 1614 of the item 204 and the dimensions 1614 of the item 204 in the encoded vector library 128.

[0221] Returning to operation 1512, the item tracking device 104 proceeds to operation 1516 in response to determining that the feature descriptors 1608 do not identify dimensions 1614 for the item 204. At operation 1516, the item tracking device 104 determines whether the feature descriptors 1608 identify a weight 1616 for the item 204. The weight 1616 identifies the weight of an item 204. The weight 1616 may be in pounds, ounces, litters, or any other suitable units.

[0222] The item tracking device 104 proceeds to operation 1518 in response to determining that the feature descriptors 1608 identify a weight 1616 for the item 204. In this case, the item tracking device 104 proceeds to operation 1518 to reduce the number of entries 1602 in the encoded vector library 128 based on the weight 1616 of the item 204.

[0223] At operation 1518, the item tracking device 104 filters the encoded vector library 128 based on the weight of the item 204. Here, the item tracking device 104 uses the weight 1616 to filter out or remove any entries 1602 in the encoded vector library 128 that do not contain the same weight 1616 as the item 204 or within a predetermined tolerance of the weight 1616 of the item 204.

[0224] In some embodiments, the item tracking device 104 may repeat a similar process to filter or reduce the number of entries 1602 in the encoded vector library 128 based on any other suitable type or combination of feature descriptors 1608.

Similarity Vectors

[0225] After filtering the encoded vector library 128 based on the feature descriptors 1608 of the item 204, the item tracking device 104 may generate a similarity vector 1704 for a received encoded vector 1702. A similarity vector 1704 comprises an array of numerical values 1710 where each numerical value 1710 indicates how similar the values in the received encoded vector 1702 are to the values in an encoded vector 1606 in the encoded vector library 128. In one embodiment, the item tracking device 104 may generate the similarity vector 1704 by using matrix multiplication between the received encoded vector 1702 and the encoded vectors 1606 in the encoded library 128. Referring to FIG. 17 as an example, the dimensions of the encoded vectors 1606 in the encoded vector library 128 may be M-by-N, where M is the number of entries 1602 in the encoded vector library 128, for example, after filtering the encoded vector library 128, and N is the length of each encoded vector 1606, which corresponds with the number of numerical values 1706 in an encoded vector 1606. The encoded vector 1702 for an unidentified item 204 may have the dimensions of N-by-1 where is N is the length of the encoded vector 1702, which corresponds with the number of numerical values 1708 in the encoded vector 1702. In this example, the item tracking device 104 may generate the similarity vector 1704 by performing matrix multiplication between the encoded vector 1702 and the encoded vectors 1606 in the encoded vector library 128. The resulting similarity vector 1704 has the dimensions of N-by-1 where N is the length of the similarity vector 1704 which is the same length as the encoded vector 1702. Each numerical value 1710 in the similarity vector 1704 corresponds with an entry 1602 in the encoded vector library 128. For example, the first numerical value 1710 in the similarity vector 1704 indicates how similar the values in the encoded vector 1702 are to the values in the encoded vector 1606 in the first entry 1602 of the encoded vector library 128, the second numerical value 1710 in the similarity vector 1704 indicates how similar the values in the encoded vector 1702 are to the values in the encoded vector 1606 in the second entry 1602 of the encoded vector library 128, and so on.

[0226] After generating the similarity vector 1704, the item tracking device 104 can identify which entry 1602, or entries 1602, in the encoded vector library 128 most closely matches the encoded vector 1702 for the identified item 204. In one embodiment, the entry 1602 that is associated with the highest numerical value 1710 in the similarity vector 1704 corresponds is the entry 1602 that closest matches the encoded vector 1702 for the item 204. After identifying the entry 1602 from the encoded vector library 128 that most closely matches the encoded vector 1702 for the identified item 204, the item tracking device 104 may then identify the item identifier 1604 that is associated with the identified entry 1602. Through this process, the item tracking device 104 is able to determine which item 204 from the encoded vector library 128 corresponds with the unidentified item 204 based on its encoded vector 1702. The item tracking device 104 then output or use the identified item identifier 1604 for other processes such as process 2300 that is described in FIG. 23.

Item Dimensioning Process Using Point Cloud Information

[0227] FIG. 18 is a flowchart of an embodiment of an item dimensioning process 1800 using point cloud information. The item tracking system 100 may employ process 1800 to determine the dimensions 1614 of an item 204 that is placed on the platform 202. This process generally involves first capturing 3D point cloud data for an item 204 using multiple 3D sensors 110 and then combining the 3D point cloud data from all of the 3D sensors 110 to generate a more complete point cloud representation of the item 204. After combining the point cloud data from the 3D sensors 110, the item tracking device 104 then determines the dimensions 1614 of the item 204 based on the new point cloud data representation. This process allows the item tracking device 104 to determine the dimensions 1614 of an item 204 without having a user take physical measurements of the item 204.

[0228] At operation 1802, the item tracking device 104 captures point cloud data 1902 of items 204 on the platform 202 using an overhead 3D sensor 110. The point cloud data 1902 comprises a plurality of data points 1901 within a 3D space. Each data point 1901 is associated with an (x, y, z) coordinate that identifies the location of the data point 1901 within the 3D space. In general, the point cloud data 1902 corresponds with the surfaces of objects that are visible to the 3D sensor 110. Referring to FIG. 19 as an example, FIG. 19 illustrates an example of point cloud data 1902 that is captured using an overhead 3D sensor 110. In this example, the 3D sensor 110 is positioned directly above the platform 202 and is configured to capture point cloud data 1902 that represents upward-facing surfaces of the items 204 on the platform 202. The 3D sensor 110 captures point cloud data 1902A that corresponds with a first item 204 and point cloud data 1902B that corresponds with a second item 204.

[0229] Returning to FIG. 18 at operation 1804, the item tracking device 104 segments the point cloud data 1902 based on clusters 1904 within the point cloud data 1902. In one embodiment, the item tracking device 104 may identify clusters 1904 within the point cloud data 1902 based on the distance between the data points 1901 in the point cloud data 1902. For example, the item tracking device 104 may use a distance threshold value to identify data points 1901 that are members of the same cluster 1904. In this example, the item tracking device 104 may compute the Euclidian distance between pairs of data points 1901 to determine whether the data points 1901 should be members of the same cluster 1904. For instance, when a pair of data points 1901 are within the distance threshold value from each other, the item tracking device 104 may associate the data points 1901 with the same cluster 1904. When the distance between a pair of data points 1901 is greater than the distance threshold value, the item tracking device 104 determines that the data points 1901 are not members of the same cluster 1904. The item tracking device 104 may repeat this process until one or more clusters 1904 have been identified within the point cloud data 1902. In other examples, the item tracking device 104 may cluster the data points 1901 using k-means clustering or any other suitable clustering technique. After identifying clusters 1904 within the point cloud data 1902, the item tracking device 104 segments the point cloud data 1902 based on the identified clusters 1904. Segmenting the point cloud data 1902 splits the data points 1901 in the point cloud data 1902 into smaller groups of point cloud data 1902 based on the identified clusters 1904. Each cluster 1904 of data points 1901 corresponds with a different item 204 that is placed on the platform 202.

[0230] At operation 1806, the item tracking device 104 selects a first item 204 from the segmented point cloud data 1902. Here, the item tracking device 104 identifies one of the items 204 on the platform 202 to begin aggregating the point cloud data 1902 from other 3D sensors 110 that are associated with the first item 204. The item tracking device 104 may iteratively select each item 204 from the platform 202. Returning to the example in FIG. 19, the item tracking device 104 may select a first item 204 that corresponds with cluster 1904A.

[0231] Returning to FIG. 18 at operation 1808, the item tracking device 104 identifies a region-of-interest 1906 for the first item 204 within the point cloud data 1902. The region-of-interest 1906 identifies a region within the 3D space. For example, the region-of-interest 1906 may define a range of x-values, y-values, and/or z-values within the 3D space. Returning to the example in FIG. 19, the item tracking device 104 may identify a region-of-interest 1906A that contains the point cloud data 1902A for the first item 204. In this example, the item tracking device 104 identifies the range of x-values, y-values, and z-values within the 3D space that contains the point cloud data 1902A.

[0232] Returning to FIG. 18 at operation 1810, the item tracking device 104 extracts point cloud data 1902 from the identified region-of-interest 1906. Here, the item tracking device 104 identifies and extracts the point cloud data 1902 from within the region-of-interest 1906 for the first item 204. By extracting the point cloud data 1902 within the region-of-interest 1906, the item tracking device 104 is able to isolate the data points 1901 for the first item 204 in the point cloud data 1902 from the data points 1901 that are associated with other items 204 on the platform 202. Returning to the example in FIG. 19, the item tracking device 104 may extract the data points 1901 (i.e. point cloud data 1902A) within the region-of-interest 1906A from the point cloud data 1902 for all the items 204 on the platform 202.

[0233] Returning to FIG. 18 at operation 1812, the item tracking device 104 selects another 3D sensor 110. After extracting point cloud data 1902 for the first item 204 from the overhead 3D sensor 110, the item tracking device 104 may repeat the same process to extract additional point cloud data 1902 for the first item 204 from the perspective of other 3D sensors 110. Each 3D sensor 110 is only able to capture point cloud data 1902 for the portion of the first item 204 that is visible to the 3D sensor 110. By capturing point cloud data 1902 from multiple 3D sensors 110 with different views of the first item 204, the item tracking device 104 is able to generate a more complete point cloud data representation of the first item 204. The item tracking device 104 may iteratively select a different 3D sensor 110 from among the 3D sensors 110 of the imaging device 102.

[0234] At operation 1814, the item tracking device 104 captures point cloud data 1902 using the selected 3D sensor 110. Here, the item tracking device 104 uses a process similar to the process described in operation 1802 to capture point cloud data 1902 using the selected 3D sensor 110. Referring to FIG. 20 as an example, the item tracking device 104 may select a 3D sensor 110 that has a side perspective view of the items 204 on the platform 202. In other words, the selected 3D sensor 110 captures point cloud data 1902 that represents side-facing surfaces of the items 204 on the platform 202. In this example, the 3D sensor 110 captures point cloud data 1902C that corresponds with the first item 204 and point cloud data 1902D that corresponds with the second item 204.

[0235] Returning to FIG. 18 at operation 1816, the item tracking device 104 identifies a region-of-interest 1906 corresponding with the first item 204 for the selected 3D sensor 110. In one embodiment, the item tracking device 104 may use a homography 608 to determine the region-of-interest 1906 for the selected 3D sensor 110 based on the region-of-interest 1906 identified by the overhead 3D sensor 110. In this case, the item tracking device 104 may identify a homography 608 that is associated with the selected 3D sensor 110. The homography 608 is configured similarly to as described in FIGS. 12A and 12B. After identifying the homography 608 that is associated with the 3D sensor 110, the item tracking device 104 uses the homography 608 to convert the range of x-values, y-values, and z-values within the 3D space that are associated with the region-of-interest 1906 for the overhead 3D sensor 110 to a corresponding range of x-values, y-values, and z-values within the 3D space that are associated with the selected 3D sensor 110. In other examples, the item tracking device 104 may use any other suitable technique for identifying a region-of-interest 1906 for the first item 204. For example, the item tracking device 104 may use a process similar to the process described in operation 1808. Returning to the example in FIG. 20, the item tracking device 104 identifies a region-of-interest 1906B that contains the point cloud data 1902C for the first item 204. In this example, the item tracking device 104 identifies the range of x-values, y-values, and z-values within the 3D space that contains the point cloud data 1902C.

[0236] Returning to FIG. 18 at operation 1818, the item tracking device 104 extracts point cloud data 1902 from the region-of-interest 1906 corresponding with the first item 204. Here, the item tracking device 104 identifies and extracts the point cloud data 1902 from within the identified region-of-interest 1906 for the first item 204. Returning to the example in FIG. 20, the item tracking device 104 may extract the data points 1901 (i.e. point cloud data 1902C) within the region-of-interest 1906B from the point cloud data 1902 for all the items 204 on the platform 202.

[0237] Returning to FIG. 18 at operation 1820, the item tracking device 104 determines whether to select another 3D sensor 110. Here, the item tracking device 104 determines whether to collect additional point cloud data 1902 for the first item 204. In one embodiment, the item tracking device 104 may determine whether to select another 3D sensor 110 based on the amount of point cloud data 1902 that has been collected. For example, the item tracking device 104 may be configured to collect point cloud data 1902 from a predetermined number (e.g. three) of 3D sensors 110. In this example, the item tracking device 104 may keep track of how many sets of point cloud data 1902 have been collected. Each set of collected point cloud data 1902 corresponds with point cloud data 1902 that has been obtained from a 3D sensor 110. The item tracking device 104 then compares the number of collected sets of point cloud data 1902 to the predetermined number of 3D sensors 110. The item tracking device 104 determines to select another 3D sensor 110 when the number of collected sets of point cloud data 1902 is less than the predetermined number of 3D sensors 110.

[0238] As another example, the item tracking device 104 may determine whether to select another 3D sensor 110 to collect additional point cloud data 1902 based on the number of data points 1901 that have been collected for the first item 204. In this example, the item tracking device 104 may determine the number of data points 1901 that have been obtained from all of the extracted point cloud data 1902 for the first item 204. The item tracking device 104 compares the number of obtained data points 1901 to a predetermined data point threshold value. The data threshold value identifies a minimum number of data points 1901 that should be collected for the first item 204. The item tracking device 104 determines to select another 3D sensor 110 when the number of collected data points 1901 is less than the predetermined data point threshold value. In other examples, the item tracking device 104 may determine whether to select another 3D sensor 110 to collect additional point cloud data 1902 based on any other suitable type of criteria.

[0239] The item tracking device 104 returns to operation 1812 in response to determining to select another 3D sensor. In this case, the item tracking device 104 returns to operation 1812 to select another 3D sensor 110 and to obtain additional point cloud data 1902 for the first item 204. Referring to FIG. 21 as an example, the item tracking device 104 may determine to select another 3D sensor 110 that has a side perspective view of the items 204 on the platform 202. In this example, the 3D sensor 110 captures point cloud data 1902E that corresponds with the first item 204 and point cloud data 1902F that corresponds with the second item 204. The item tracking device 104 then identifies a region-of-interest 1906C that contains the point cloud data 1902E for the first item 204. In this example, the item tracking device 104 identifies the range of x-values, y-values, and z-values within the 3D space that contains the point cloud data 1902E. After identifying the region-of-interest 1906C, the item tracking device 104 extracts the data points 1901 (i.e. point cloud data 1902E) within the region-of-interest 1906C from the point cloud data 1902 for all the items 204 on the platform 202. The item tracking device 104 may repeat this process for any other selected 3D sensors 110.

[0240] Returning to FIG. 18 at operation 1820, the item tracking device 104 proceeds to operation 1822 in response to determining to not select another 3D sensor 110. At operation 1822, the item tracking device 104 combines the extracted point cloud data 1902 for the first item 204. Here, the item tracking device 104 merges all of the collected point cloud data 1902 into a single set of point cloud data 1902. By combining the point cloud data 1902 from multiple 3D sensors 110, the item tracking device 104 can generate a more complete point cloud data representation of the first item 204 that can be used for determining the dimensions 1614 of the first item 204. Referring to FIG. 22 as an example, the item tracking device 104 may combine point cloud data 1902A, 1902C, and 1902E into a single set of point cloud data 1902G. The combined point cloud data 1902G contains all of the data points 1901 from point cloud data 1902A, 1902C, and 1902E.

[0241] Returning to FIG. 18 at operation 1824, the item tracking device 104 determines the dimensions 1614 of the first item 204 based on the combined point cloud data 1902. In one embodiment, the item tracking device 104 may determine the dimensions 1614 of the item 204 by determining the distance between data points 1901 at the edges of the combined point cloud data 1902. For example, the item tracking device 104 may identify a pair of data points 1901 on opposing ends of the combined point cloud data 1902 and then compute the distance (e.g. Euclidean distance) between the pair of data points 1901. In this example, the distance between the data points 1901 may correspond with the length 2202, width 2204, or height 2206 of the first item 204. In other examples, the item tracking device 104 may determine the dimensions 1614 of the first item 204 using any other suitable technique. Returning to the example in FIG. 22, the item tracking device 104 may determine a length 2202, a width 2204, and a height 2206 for the first item 204 based on the combined point cloud data 1902G.

[0242] Returning to FIG. 18 at operation 1826, the item tracking device 104 determines whether to determine the dimensions 1614 for another item 204. In one embodiment, the item tracking device 104 may be configured to determine the dimensions 1614 for all of the items 204 that are on the platform 202. In this case, the item tracking device 104 may determine whether the dimensions 1614 for all of the items 204 on the platform 202 have been determined. The item tracking device 104 will determine the dimensions 1614 for another item 204 when the dimensions 1614 of an item 204 are still unknown and have not yet been determined. In other examples, the item tracking device 104 may determine whether to determine the dimensions 1614 for another item 204 based on any other suitable criteria.

[0243] The item tracking device 104 returns to operation 1806 in response to determining to find the dimensions 1614 for another item 204. In this case, the item tracking device 104 returns to operation 1806 to collect point cloud data 1902 for a different item 204. The item tracking device 104 may then repeat the same process of aggregating point cloud data 1902 from multiple 3D sensors 110, combining the point cloud data 1902, and then determining the dimensions 1614 of the item 204 based on the combined point cloud data 1902.

[0244] In response to determining not to determine the dimensions 1614 for another item 204, the item tracking device 104 may store the dimensions 1614 for the first item 204. For example, the item tracking device 104 may obtain an item identifier 1604 for the first item 204 and then generate an entry 1602 in the encoded vector library 128 that associates the determined length 2202, width 2204, and height 2206 with the first item 204 as feature descriptors 1608. In some embodiments, the item tracking device 104 may store the length 2202, width 2204, and height 2206 for the first item 204 in ascending order when generating the entry 1602.

[0245] In other embodiments, the item tracking device 104 may output or store the determined length 2202, width 2204, and height 2206 for the first item 204 as feature descriptors 1608 for other processes such as item identification. For instance, the item tracking device 104 may use the feature descriptors 1608 to help identify the first item 204 using a process similar to process 2300 that is described in FIG. 23.

An Item Tracking Process Using Encoded Vectors

[0246] FIG. 23 is a flowchart of an embodiment of an item tracking process 2300 for using encoded vectors 1606 for the item tracking system 100. The item tracking system 100 may employ process 2300 to identify items 204 that are placed on the platform 202 of an imaging device 102 and to assign the items 204 to a particular user. As an example, the item tracking system 100 may employ process 2300 within a store to add items 204 to a user's digital cart for purchase. As another example, the item tracking system 100 may employ process 2300 within a warehouse or supply room to check out items to a user. In other examples, the item tracking system 100 may employ process 2300 in any other suitable type of application where items 204 are assigned or associated with a particular user. This process allows the user to obtain items 204 from a space without having the user scan or otherwise identify the items 204 they would like to take.

[0247] At operation 2302, the item tracking device 104 performs auto-exclusion for the imaging device 102. The item tracking device 104 may perform auto-exclusion using a process similar to the process described in operation 302 of FIG. 3. For example, during an initial calibration period, the platform 202 may not have any items 204 placed on the platform 202. During this period of time, the item tracking device 104 may use one or more cameras 108 and/or 3D sensors 110 to capture reference images 122 and reference depth images 124, respectively, of the platform 202 without any items 204 placed on the platform 202. The item tracking device 104 can then use the captured images 122 and depth images 124 as reference images to detect when an item 204 is placed on the platform 202. At a later time, the item tracking device 104 can detect that an item 204 has been placed on the surface 208 of the platform 202 based on differences in depth values between subsequent depth images 124 and the reference depth image 124 and/or differences in the pixel values between subsequent images 122 and the reference image 122.

[0248] At operation 2304, the item tracking device 104 determines whether a hand has been detected above the platform 202. In one embodiment, the item tracking device 104 may use a process similar to process 700 that is described in FIG. 7 for detecting a triggering event that corresponds with a user's hand being detected above the platform 202. For example, the item tracking device 104 may check for differences between a reference depth image 124 and a subsequent depth image 124 to detect the presence of an object above the platform 202. The item tracking device 104 then checks whether the object corresponds with a user's hand or an item 204 that is placed on the platform 202. The item tracking device 104 determines that the object is a user's hand when a first portion of the object (e.g., a user's wrist or arm) is outside a region-of-interest 802 for the platform 202 and a second portion of the object (e.g., a user's hand) is within the region-of-interest 802 for the platform 202. When this condition is met, the item tracking device 104 determines that a user's hand has been detected above the platform 202. In other examples, the item tracking device 104 may use proximity sensors, motion sensors, or any other suitable technique for detecting whether a user's hand has been detected above the platform 202.

[0249] The item tracking device 104 remains at operation 2304 in response to determining that a user's hand has not been detected above the platform 202. In this case, the item tracking device 104 remains at operation 2304 to keep checking for the presence of a user's hand as a triggering event. The item tracking device 104 proceeds to operation 2306 in response to determining that a user's hand has been detected. In this case, the item tracking device 104 uses the presence of a user's hand as a triggering event and proceeds to operation 2306 to begin identifying any items 204 that the user has placed on the platform 202.

[0250] At operation 2306, the item tracking device 104 performs segmentation using an overhead view of the platform 202. In one embodiment, the item tracking device 104 may perform segmentation using a depth image 124 from a 3D sensor 110 that is configured with overhead or perspective view of the items 204 on the platform 202. In this example, the item tracking device 104 captures an overhead depth image 124 of the items 204 that are placed on the platform 202. The item tracking device 104 may then use a depth threshold value to distinguish between the platform 202 and items 204 that are placed on the platform 202 in the captured depth image 124. For instance, the item tracking device 104 may set a depth threshold value that is just above the surface of the platform 202. This depth threshold value may be determined based on the pixel values corresponding with the surface of the platform 202 in the reference depth images 124 that were captured during the auto-exclusion process in operation 2302. After setting the depth threshold value, the item tracking device 104 may apply the depth threshold value to the captured depth image 124 to filter out or remove the platform 202 from the depth image 124. After filtering the depth image 124, the remaining clusters of pixels correspond with items 204 that are placed on the platform 202. Each cluster of pixels corresponds with a different item 204. After identifying the clusters of pixels for each item 204, the item tracking device 104 then counts the number of items 204 that are placed on the platform 202 based on the number of pixel clusters that are present in the depth image 124. This number of items 204 is used later to determine whether all of items 204 on the platform 202 have been identified.

[0251] At operation 2308, the item tracking device 104 captures images 122 of the items 204 on the platform 202. Here, the item tracking device 104 captures multiple images 122 of the items 204 on the platform 202 using multiple cameras 108. For example, the item tracking device 104 may capture images 122 with an overhead view, a perspective view, and/or a side view of the items 204 on the platform 202. The item tracking device 104 may also capture multiple depth images 124 of the items 204 on the platform 202 using one or more 3D sensors 110.

[0252] At operation 2310, the item tracking device 104 generates cropped images 122 of the items 204 in each image 122. In one embodiment, the item tracking device 104 generates a cropped image 122 of an item 204 based on the features of the item 204 that are present in an image 122. The item tracking device 104 may first identify a region-of-interest (e.g., a bounding box) for an item 204 based on the detected features of the item 204 that are present in an image 122 and then may crop the image 122 based on the identified region-of-interest. The region-of-interest comprises a plurality of pixels that correspond with the item 204 in a captured image 122 or depth image 124 of the item 204 on the platform 202. The item tracking device 104 may employ one or more image processing techniques to identify a region-of-interest for an item 204 within an image 122 based on the features and physical attributes of the item 204. For example, the item tracking device 104 may employ object detection and/or OCR to identify text, logos, branding, colors, barcodes, or any other features of an item 204 that can be used to identify the item 204. In this case, the item tracking device 104 may process pixels within an image 122 to identify text, colors, barcodes, patterns, or any other characteristics of an item 204. The item tracking device 104 may then compare the identified features of the item 204 to a set of features that correspond with different items 204. For instance, the item tracking device 104 may extract text (e.g. a product name) from an image 122 and may compare the text to a set of text that is associated with different items 204. As another example, the item tracking device 104 may determine a dominant color within an image 122 and may compare the dominant color to a set of colors that are associated with different items 204. As another example, the item tracking device 104 may identify a barcode within an image 122 and may compare the barcode to a set of barcodes that are associated with different items 204. As another example, the item tracking device 104 may identify logos or patterns within the image 122 and may compare the identified logos or patterns to a set of logos or patterns that are associated with different items 204. In other examples, the item tracking device 104 may identify any other suitable type or combination of features and compare the identified features to features that are associated with different items 204.

[0253] After comparing the identified features of the item 204 to the set of features that are associated with different items 204, the item tracking device 104 then determines whether a match is found. The item tracking device 104 may determine that a match is found when at least a meaningful portion of the identified features match features that correspond with an item 204. In response to determining that a meaningful portion of features within an image 122 match the features of an item 204, the item tracking device 104 may identify a region-of-interest that corresponds with the matching item 204.

[0254] After identifying a region-of-interest for the item 204, the item tracking device 104 crops the image 122 by extracting the pixels within the region-of-interest for the item 204 from the image 122. By cropping the image 122, the item tracking device 104 generates a second image 122 that comprises the extracted pixels within the region-of-interest for the item 204 from the original image 122. This process allows the item tracking device 104 to generate a new image 122 that contains an item 204 that is on the platform 202. The item tracking device 104 repeats this process for all of the items 204 within a captured image 122 and all of the captured images 122 of the items 204 on the platform 202. The result of this process is a set of cropped images 122 that each correspond with an item 204 that is placed on the platform 202.

[0255] In some embodiments, the item tracking device 104 may use a process similar to process 900 in FIG. 9 to generate the cropped images 122 of the items 204. In some embodiments, operation 2310 may be optional and omitted. For example, operation 2310 may be omitted when the item tracking device 104 detects that only one item 204 is placed on the platform 202.

[0256] At operation 2312, the item tracking device 104 obtains an encoded vector 1606 for each item 204. An encoded vector 1606 comprises an array of numerical values. Each numerical value in the encoded vector 1606 corresponds with and describes an attribute (e.g., item type, size, shape, color, etc.) of an item 204. An encoded vector 1606 may be any suitable length. The item tracking device 104 obtains an encoded vector 1606 for each item 204 by inputting each of the images 122 (e.g., cropped images 122) from operation 2310 into the machine learning model 126. The machine learning model 126 is configured to output an encoded vector 1606 for an item 204 based on the features or physical attributes of the item 204 that are present in the image 122 of the item 204. Examples of physical attributes include, but are not limited to, an item type, a size, shape, color, or any other suitable type of attribute of the item 204. After inputting the image 122 of the item 204 into the machine learning model 126, the item tracking device 104 receives an encoded vector 1606 for the item 204. The item tracking device 104 repeats this process to obtain an encoded vector 1606 for each item 204 on the platform 202.

[0257] At operation 2314, the item tracking device 104 identifies each item 204 in the encoded vector library 128 based on their corresponding encoded vector 1606. Here, the item tracking device 104 uses the encoded vector 1606 for each item 204 to identify the closest matching encoded vector 1606 in the encoded vector library 128. In some embodiments, the item tracking device 104 may first reduce the search space within the encoded vector library 128 before attempting to identify an item 204. In this case, the item tracking device 104 may obtain or identify feature descriptors 1608 for the item 204 using a process similar to the process described in operation 1104 of FIG. 11. Each of the feature descriptors 1608 describes the physical characteristics of an item 204. Examples of feature descriptors 1608 include, but are not limited to, an item type 1610, a dominant color 1612, dimensions 1614, weight 1616, or any other suitable type of descriptor that describes an item 204. The item tracking device 104 may employ object detection and/or OCR to identify text, logos, branding, colors, barcodes, or any other features of an item 204 that can be used to identify the item 204. The item tracking device 104 may determine the dimensions of the item 204 using a process similar to process 1800 that is described in FIG. 18. The item tracking device 104 may determine the weight of the item 204 using a weight sensor 112. In other embodiments, the item tracking device 104 may use any other suitable process for determining feature descriptors 1608 for the item 204. After obtaining feature descriptor 1608 for an item 204, the item tracking device 104 may filter or remove the entries 1602 from consideration in the encoded vector library 128 using a process similar to process 1500 in FIG. 15. After filtering the entries 1602 in the encoded vector library 128, the item tracking device 104 may then identify the closest matching encoded vector 1606 in the encoded vector library 128 to the encoded vector 1606 for an unidentified item 204. This process reduces the amount of time required to search for a corresponding entry 1602 in the encoded vector library 128 as well as improves the accuracy of the results from identifying an entry 1602 in the encoded vector library 128.

[0258] In one embodiment, the item tracking device 104 identifies the closest matching encoded vector 1606 in the encoded vector library 128 by generating a similarity vector 1704 between the encoded vector 1606 for an unidentified item 204 and the remaining encoded vectors 1606 in the encoded vector library 128. The similarity vector 1704 comprises an array of numerical values 1710 where each numerical value 1710 indicates how similar the values in the encoded vector 1606 for the item 204 are to the values in an encoded vector 1606 in the encoded vector library 128. In one embodiment, the item tracking device 104 may generate the similarity vector 1704 by using a process similar to the process described in FIG. 17. In this example, the item tracking device 104 uses matrix multiplication between the encoded vector 1606 for the item 204 and the encoded vectors 1606 in the encoded vector library 128. Each numerical value 1710 in the similarity vector 1704 corresponds with an entry 1602 in the encoded vector library 128. For example, the first numerical value 1710 in the similarity vector 1704 indicates how similar the values in the encoded vector 1702 are to the values in the encoded vector 1606 in the first entry 1602 of the encoded vector library 128, the second numerical value 1710 in the similarity vector 1704 indicates how similar the values in the encoded vector 1702 are to the values in the encoded vector 1606 in the second entry 1602 of the encoded vector library 128, and so on.

[0259] After generating the similarity vector 1704, the item tracking device 104 can identify which entry 1602, or entries 1602, in the encoded vector library 128 most closely matches the encoded vector 1606 for the item 204. In one embodiment, the entry 1602 that is associated with the highest numerical value 1710 in the similarity vector 1704 corresponds is the entry 1602 that most closely matches the encoded vector 1606 for the item 204. After identifying the entry 1602 from the encoded vector library 128 that most closely matches the encoded vector 1606 for the item 204, the item tracking device 104 may then identify the item identifier 1604 from the encoded vector library 128 that is associated with the identified entry 1602. Through this process, the item tracking device 104 is able to which item 204 from the encoded vector library 128 corresponds with the item 204 based on its encoded vector 1606. The item tracking device 104 then outputs the identified item identifier 1604 for the identified item 204. For example, the item tracking device 104 may output the identified item identifier 1604 for the identified item 204 by adding the item identifier 1604 to a list of identified items 204 that is on a graphical user interface. The item tracking device 104 repeats this process for all of the encoded vectors 1606 that were obtained in operation 2312.

[0260] At operation 2316, the item tracking device 104 determines whether all of the items 204 have been identified. Here, the item tracking device 104 determines whether the number of identified items 204 matches the number of items 204 that were detected on the platform 202 in operation 2306. The item tracking device 104 determines that all of the items 204 have been identified when the number of identified items 204 matches the number of items 204 that were detected on the platform 202. Otherwise, the item tracking device 104 determines that one or more items 204 have not been identified when the number of identified items 204 does not match the number of items 204 that were detected on the platform 202.

[0261] The item tracking device 104 proceeds to operation 2318 in response to determining that one or more items 204 have not been identified. In this case, the item tracking device 104 proceeds to operation 2318 to ask the user to identify the one or more items 204 that have not been identified. At operation 2318, the item tracking device 104 outputs a prompt requesting the user to identify one or more items 204 on the platform 202. In one embodiment, the item tracking device 104 may request for the user to identify an item 204 from among a set of similar items 204. Referring to FIG. 24 as an example, the item tracking device 104 may output a screen 2400 that displays items 204 that were detected (shown as display elements 2402) as well as any items 204 that were not identified. In this example, the screen 2400 displays the recommendations (shown as display elements 2404) for other similar items 204 in the event that an item 204 is not identified. In one embodiment, the item recommendations may correspond with other items 204 that were identified using the similarity vector 1704. For example, the item recommendations may comprise items 204 that are associated with the second and third highest values in the similarity vector 1704. The user may provide a user input to select the any items 204 that were not identified.

[0262] In some embodiments, the item tracking device 104 may prompt the user scan any items 204 that were not identified. For example, the item tracking device 104 may provide instructions for the user to scan a barcode of an item 204 using a barcode scanner. In this case, the item tracking device 104 may use the graphical user interface to display a combination of items 204 that were detected on the platform 202 as well as items 204 that were manually scanned by the user. Referring to FIG. 25 as an example, the item tracking device 104 may output a screen 2500 that displays items 204 (shown as display elements 2502) that were detected on the platform 202 and items 204 (shown as display elements 2504) that were manually scanned by the user.

[0263] Returning to FIG. 23 at operation 2316, the item tracking device 104 proceeds to operation 2320 in response to determining that all of the items 204 have been identified. At operation 2320, the item tracking device 104 determines whether there are any additional items 204 to detect for the user. In some embodiments, the user may provide a user input that indicates that the user would like to add additional items 204 to the platform 202. In other embodiments, the item tracking device 104 may use the presence of the user's hand removing and adding items 204 from the platform 202 to determine whether there are additional items 204 to detect for the user. The item tracking device 104 returns to operation 2304 in response to determining that there are additional items 204 to detect. In this case, the item tracking device 104 returns to operation 2304 to begin detecting additional items 204 that the user places on the platform 202. The item tracking device 104 proceeds to operation 2322 in response to determining that there are no additional items 204 to detect for the user. In this case, the item tracking device 104 proceeds to operation 2322 to associate the detected items 204 with the user.

[0264] Before associating the items 204 with the user, the item tracking device 104 may allow the user to remove one or more items 204 from the list of identified items 204 by selecting the items 204 on the graphical user interface. Referring to FIG. 26 as an example, the item tracking device 104 may receive a user input that identifies an item 204 to remove from the list of identified items 204 and output a screen 2600 that confirms that the user would like to remove the item 204. This feature allows the user to edit and finalize the list of detected items 204 that they would like to purchase.

[0265] Returning to FIG. 23 at operation 2322, the item tracking device 104 associates the items 204 with the user. In one embodiment, the item tracking device 104 may identify the user that placed the items 204 on the platform 202. For example, the user may identify themselves using a scanner or card reader that is located at the imaging device 102. Examples of a scanner include, but are not limited to, a QR code scanner, a barcode scanner, an NFC scanner, or any other suitable type of scanner that can receive an electronic code embedded with information that uniquely identifies a person. In other examples, the user may identify themselves by providing user information on a graphical user interface that is located at the imaging device 102. Examples of user information include, but are not limited to, a name, a phone number, an email address, an identification number, an employee number, an alphanumeric code, or any other suitable type of information that is associated with the user.

[0266] The item tracking device 104 uses the information provided by the user to identify an account that is associated with the user and then to add the identified items 204 to the user's account. For example, the item tracking device 104 may use the information provided by the user to identify an account within the user account information 120 that is associated with the user. As an example, the item tracking device 104 may identify a digital cart that is associated with the user. In this example, the digital cart comprises information about items 204 that the user has placed on the platform 202 to purchase. The item tracking device 104 may add the items 204 to the user's digital cart by adding the item identifiers 1604 for the identified items 204 to the digital cart. The item tracking device 104 may also add other information to the digital cart that is related to the items 204. For example, the item tracking device 104 may use the item identifiers 1604 to look up pricing information for the identified items 204 from the stored item information 118. The item tracking device 104 may then add pricing information that corresponds with each of the identified items 204 to the user's digital cart.

[0267] After the item tracking device 104 adds the items 204 to the user's digital cart, the item tracking device 104 may trigger or initiate a transaction for the items 204. In one embodiment, the item tracking device 104 may use previously stored information (e.g. payment card information) to complete the transaction for the items 204. In this case, the user may be automatically charged for the items 204 in their digital cart when they leave the space. In other embodiments, the item tracking device 104 may collect information from the user using a scanner or card reader that is located at the imaging device 102 to complete the transaction for the items 204. This process allows the items 204 to be automatically added to the user's account (e.g. digital cart) without having the user scan or otherwise identify the items 204 they would like to take. After adding the items 204 to the user's account, the item tracking device 104 may output a notification or summary to the user with information about the items 204 that were added to the user's account. For example, the item tracking device 104 may output a summary on a graphical user interface that is located at the imaging device 102. As another example, the item tracking device 104 may output a summary by sending the summary to an email address or a user device that is associated with the user.

Identifying an Item Based on Associations with Other Items

[0268] In some cases, item tracking device 104 may be unable to identify an item 204 placed on the platform 202. In such cases, as described further below, item tracking device 104 may identify the unidentified item 204 based on a pre-defined association 2802 (shown in FIG. 28) between the unidentified item 204 and another item 204 on the platform 202 that was previously identified as part of the same transaction. For example, as shown in FIG. 27, a transaction may include placement of a first item 204A (e.g., a 1-liter bottle of soda) on the platform 202. Item tracking device 104 may successfully identify the first item 204A as a 1-liter bottle of soda and assign a corresponding item identifier 1604a (shown as I2 in FIG. 28) from the encoded vector library 128. Item tracking device 104 may use a process similar to process 2300 that is described with reference to FIG. 23 to identify the first item 204A. For example, as described with reference to FIG. 29, identifying the first item 204A includes generating cropped images 2702 of the first item 204A, wherein the first item 204A is identified based on the cropped images 2702 of the first item 204A. Once the first item 204A is identified, a second item 204B (e.g., a small bag of chips) may be subsequently placed on the platform 202 as part of the same transaction. In one embodiment, the placement of the first item 204A may be referred to as a first interaction of the transaction and the placement of the second item 204B may be referred to as a second interaction of the same transaction. In some embodiments, item tracking device 104 may be unable to identify the second item 204b. In such a case, as described further below with reference to FIG. 29, item tracking device 104 may identify the second item 204B based on a pre-defined association 2802 between the unidentified second item 204B and the previously identified first item 204A. As described with reference to FIG. 29, identifying the second item 204B includes generating cropped images 2704 of the second item 204B, wherein the second item 204B is identified based on the cropped images 2704 of the second item 204B.

[0269] In this context, referring to FIG. 28, item tracking device 104 stores (e.g., in memory 116) associations 2802 between item identifiers 1604 of items 204 listed in the encoded vector library 128. An association 2802 between two item identifiers 1604 may correspond to any logical association between items 204 associated with the item identifiers 1604. For example, when the item tracking system 100 is deployed and used in a store where a plurality of items 204 are available for purchase, the store may offer certain promotions when two or more items 204 are purchased together in a single transaction. One example promotion may include a small bag of chips free with the purchase of a 1-liter bottle of soda. Another example promotion may include a reduced price or “buy one get one free” when two 16 oz soda bottles of a particular brand and/or flavor are purchased together in a single transaction. In such cases, a particular promotion that includes two or more items 204 may be stored as an association 2802 between the respective item identifiers 1604 (e.g., stored in the encoded vector library 128) of the items 204. It may be noted that an association 2802 between two item identifiers 1604 may include an association between two instances of the same item identifier 1604 associated with the same item 204. For example, when an example promotion includes two or more of the same item 204 (e.g., buy two of the same item 204 for a reduced price), this promotion is stored in the memory as an association 2802 between two or more instances of the same item identifier 1604 associated with the same item 204.

[0270] As shown in FIG. 28 associations 2802 are stored (e.g., in memory 116) as part of the encoded vector library 128. As shown, each of the entries 1602 is associated with an association 2802. Entry 1602a is associated with association-1 (shown as A1), entries 1602b and 1602c are associated with association-2 (shown as A2), and entry 1602d is associated with association-3 (shown as A3). In one example, association-1 may indicate a promotion associated with two or more of the same items 204 having the same item identifier 1604 (shown as I1) stored in entry 1602a. For example, association-1 may indicate a promotion including a reduced price when two 16 oz water bottles of the same brand are purchased together as part of the same transaction. In this example, the 16 oz water bottle is associated with the item identifier 1604 (I1) from entry 1602a. Similarly, association-3 may indicate a promotion associated with two or more of the same items 204 having the same item identifier 1604 (shown as I4) stored in entry 1602d. For example, association-3 may indicate a promotion including a reduced price when two 16 oz bottles of soda of the same brand and/or flavor are purchased together as part of the same transaction. In this example, the 16 oz bottle of soda is associated with the item identifier 1604 (I4) from entry 1602d. Association-2, for example, may indicate a promotion associated with two different items 204 having two different item identifiers 1604a (I2) and 1604b (I3) stored in the respective entries 1602b and 1602c. For example, association-2 may indicate a promotion including a bag of chips free with a 1-liter bottle of soda. In this example, the 1-liter bottle of soda may be associated with a first item identifier 1604a (I2) from entry 1602b and the bag of chips may be associated with a second item identifier 1604b (I3) from entry 1602c.

[0271] FIG. 29 illustrates a flowchart of an example method 2900 for identifying a second item 204B based on an association 2802 with a first item 204A, in accordance with one or more embodiments of the present disclosure. Method 2900 may be performed by item tracking device 104 as shown in FIG. 1.

[0272] At operation 2902, item tracking device 104 detects a first triggering event at the platform 202, wherein the first triggering event corresponds to the placement of a first item 204A on the platform 202. In a particular embodiment, the first triggering event may correspond to a user placing the first item 204A on the platform 202.

[0273] As described above, the item tracking device 104 performs auto-exclusion for the imaging device 102 using a process similar to the process described in operation 302 of FIG. 3. For example, during an initial calibration period, the platform 202 may not have any items 204 placed on the platform 202. During this period of time, the item tracking device 104 may use one or more cameras 108 and/or 3D sensors 110 to capture reference images 122 and reference depth images 124, respectively, of the platform 202 without any items 204 placed on the platform 202. The item tracking device 104 can then use the captured images 122 and depth images 124 as reference images to detect when an item 204 is placed on the platform 202. At a later time, the item tracking device 104 can detect that an item 204 has been placed on the surface 208 of the platform 202 based on differences in depth values between subsequent depth images 124 and the reference depth image 124 and/or differences in the pixel values between subsequent images 122 and the reference image 122.

[0274] In one embodiment, to detect the first triggering event, the item tracking device 104 may use a process similar to process 700 that is described in FIG. 7 for detecting a triggering event, such as, for example, an event that corresponds with a user's hand being detected above the platform 202 and placing an item 204 on the platform. For example, the item tracking device 104 may check for differences between a reference depth image 124 and a subsequent depth image 124 to detect the presence of an object above the platform 202. The item tracking device 104 then checks whether the object corresponds with a user's hand or an item 204 that is placed on the platform 202. The item tracking device 104 determines that the object is a user's hand when a first portion of the object (e.g., a user's wrist or arm) is outside a region-of-interest 802 for the platform 202 and a second portion of the object (e.g., a user's hand) is within the region-of-interest 802 for the platform 202. When this condition is met, the item tracking device 104 determines that a user's hand has been detected above the platform 202. In other examples, the item tracking device 104 may use proximity sensors, motion sensors, or any other suitable technique for detecting whether a user's hand has been detected above the platform 202. After detecting the user's hand, the item tracking device 104 begins periodically capturing additional overhead depth images 124 of the platform 202 to check whether a user's hand has exited the platform 202. In response to determining that the user's hand is no longer on the platform 202, the item tracking device 104 determines whether the first item 204A is on the platform 202. In response to determining that the first item 204A has been placed on the platform, the item tracking device 104 determines that the first triggering event has occurred and proceeds to identify the first item 204A that the user has placed on the platform 202.

[0275] Once the first triggering event is detected, the item tracking device 104 performs segmentation using an overhead view of the platform 202. In one embodiment, the item tracking device 104 may perform segmentation using a depth image 124 from a 3D sensor 110 that is positioned for an overhead or perspective view of the items 204 on the platform 202. In this example, the item tracking device 104 captures an overhead depth image 124 of the items 204 that are placed on the platform 202. The item tracking device 104 may then use a depth threshold value to distinguish between the platform 202 and items 204 that are placed on the platform 202 in the captured depth image 124. For instance, the item tracking device 104 may set a depth threshold value that is just above the surface of the platform 202. This depth threshold value may be determined based on the pixel values corresponding with the surface of the platform 202 in the reference depth images 124 that were captured during the auto-exclusion process described above. After setting the depth threshold value, the item tracking device 104 may apply the depth threshold value to the captured depth image 124 to filter out or remove the platform 202 from the depth image 124. After filtering the depth image 124, the remaining clusters of pixels correspond with items 204 that are placed on the platform 202. Each cluster of pixels corresponds with a different item 204. For example, one of the clusters of pixels corresponds to the first item 204 placed on the platform 202 as part of the first triggering event detected in operation 2902.

[0276] At operation 2904, in response to detecting the first triggering event, item tracking device 104 captures a plurality of first images 122A of the first item 204 placed on the platform 202 using two or more cameras 108.

[0277] As described above, the item tracking device 104 may capture a plurality of first images 122A (as shown in FIG. 5A) of the first item 204 on the platform 202 using multiple cameras 108. For example, the item tracking device 104 may capture first images 122A with an overhead view, a perspective view, and/or a side view of the first item 204 on the platform 202.

[0278] At operation 2906, item tracking device 104 identifies a first item identifier 1604a associated with the first item 204 based on the plurality of first images 122A.

[0279] The item tracking device 104 may use a process similar to process 2300 that is described with reference to FIG. 23 to identify first item 204A. For example, the item tracking device 104 may generate a cropped image 2702 of the first item 204A from each first image 122A of the first item 204A captured by a respective camera 108 by isolating at least a portion of the first item 204A from the first image 122A. In other words, item tracking device 104 generates one cropped image 2702 of the first item 204A based on each first image 122A of the first item 204A captured by a respective camera 108. As shown in FIG. 27, item tracking device 104 generates three cropped images 2702a, 2702b and 2702c of the first item 204A from respective first images 122A of the first item 204A.

[0280] As described above, in one embodiment, the item tracking device 104 may generate a cropped image 2702 of the first item 204A based on the features of the first item 204A that are present in a first image 122A (e.g., one of the first images 122A). The item tracking device 104 may first identify a region-of-interest (e.g., a bounding box) 1002 (as shown in FIG. 10A) for the first item 204A based on the detected features of the first item 204A that are present in a first image 122A and then may crop the first image 122A based on the identified region-of-interest 1002. The region-of-interest 1002 comprises a plurality of pixels that correspond with the first item 204A in a captured first image 122A of the first item 204A on the platform 202. The item tracking device 104 may employ one or more image processing techniques to identify a region-of-interest 1002 for the first item 204A within the first image 122A based on the features and physical attributes of the first item 204A. After identifying a region-of-interest 1002 for the first item 204A, the item tracking device 104 crops the first image 122A by extracting the pixels within the region-of-interest 1002 that correspond to the first item 204A in the first image 122A. By cropping the first image 122A, the item tracking device 104 generates another image (e.g., cropped image 2702) that comprises the extracted pixels within the region-of-interest 1002 for the first item 204A from the original first image 122A. The item tracking device 104 may repeat this process for all of the captured first images 122A of the first item 204A on the platform 202. The result of this process is a set of cropped images 2702 corresponding to the first item 204A that is placed on the platform 202. In some embodiments, the item tracking device 104 may use a process similar to process 900 in FIG. 9 to generate the cropped images 2702 of the first item 204A.

[0281] The item tracking device 104 generates an encoded vector 1702 (shown in FIG. 17) for each cropped image 2702 of the first item 204A. An encoded vector 1702 comprises an array of numerical values. Each numerical value in the encoded vector 1702 corresponds with and describes an attribute (e.g., item type, size, shape, color, etc.) of the first item 204A. An encoded vector 1702 may be any suitable length. The item tracking device 104 generates an encoded vector 1702 for the first item 204A by inputting each of the cropped images 2702 into a machine learning model (e.g., machine learning model 126). The machine learning model 126 is configured to output an encoded vector 1702 for an item 204 based on the features or physical attributes of the item 204 that are present in the image 122 of the item 204. Examples of physical attributes include, but are not limited to, an item type, a size, shape, color, or any other suitable type of attribute of the item 204. After inputting a cropped image 2702 of the first item 204A into the machine learning model 126, the item tracking device 104 receives an encoded vector 1702 for the first item 204A. The item tracking device 104 repeats this process to obtain an encoded vector 1702 for each cropped image 2702 of the first item 204A on the platform 202.

[0282] The item tracking device 104 identifies the first item 204A from the encoded vector library 128 based on the corresponding encoded vector 1702 generated for the first item 204A. Here, the item tracking device 104 uses the encoded vector 1702 for the first item 204A to identify the closest matching encoded vector 1606 in the encoded vector library 128. In one embodiment, the item tracking device 104 identifies the closest matching encoded vector 1606 in the encoded vector library 128 by generating a similarity vector 1704 (shown in FIG. 17) between the encoded vector 1702 generated for the unidentified first item 204A and the encoded vectors 1606 in the encoded vector library 128. The similarity vector 1704 comprises an array of numerical similarity values 1710 where each numerical similarity value 1710 indicates how similar the values in the encoded vector 1702 for the first item 204A are to a particular encoded vector 1606 in the encoded vector library 128. In one embodiment, the item tracking device 104 may generate the similarity vector 1704 by using a process similar to the process described in FIG. 17. In this example, the item tracking device 104 uses matrix multiplication between the encoded vector 1702 for the first item 204A and the encoded vectors 1606 in the encoded vector library 128. Each numerical similarity value 1710 in the similarity vector 1704 corresponds with an entry 1602 in the encoded vector library 128. For example, the first numerical value 1710 in the similarity vector 1704 indicates how similar the values in the encoded vector 1702 are to the values in the encoded vector 1606 in the first entry 1602 of the encoded vector library 128, the second numerical value 1710 in the similarity vector 1704 indicates how similar the values in the encoded vector 1702 are to the values in the encoded vector 1606 in the second entry 1602 of the encoded vector library 128, and so on.

[0283] After generating the similarity vector 1704, the item tracking device 104 can identify which entry 1602, in the encoded vector library 128, most closely matches the encoded vector 1702 for the first item 204A. In one embodiment, the entry 1602 that is associated with the highest numerical similarity value 1710 in the similarity vector 1704 is the entry 1602 that most closely matches the encoded vector 1702 for the first item 204A. After identifying the entry 1602 from the encoded vector library 128 that most closely matches the encoded vector 1702 for the first item 204A, the item tracking device 104 may then identify the item identifier 1604 from the encoded vector library 128 that is associated with the identified entry 1602. Through this process, the item tracking device 104 is able to determine which item 204 from the encoded vector library 128 corresponds with the unidentified first item 204A based on its encoded vector 1702. The item tracking device 104 then outputs the identified item identifier 1604 for the identified item 204. The item tracking device 104 repeats this process for each encoded vector 1702 generated for each cropped image 2702 (e.g., 2702a, 2702b and 2702c) of the first item 204A. This process may yield a set of item identifiers 1604 corresponding to the first item 204A, wherein the set of item identifiers 1604 corresponding to the first item 204A may include a plurality of item identifiers 1604 corresponding to a plurality of cropped images 2702 of the first item 204A. In other words, item tracking device 104 identifies an item identifier 1604 for each cropped image 2702 of the first item 204A.

[0284] Item tracking device 104 may select one of a plurality of item identifiers 1604 identified for the first item 204A based on the respective plurality of cropped images 2702 of the first item 204A. For example, item tracking device 104 may select the first item identifier 1604a associated with the first item 204A based a plurality of item identifiers 1604 identified for the first item 204A based on the respective plurality of cropped images 2702 of the first item 204A.

[0285] In one or more embodiments, item tracking device 104 may input each cropped image 2702 of the first item 204A into a machine learning model which is configured to determine whether the cropped image 2702 of the first item 204A is a front image 122 of the first item 204A or a back image 122 of the first item 204A. A front image 122 of the first item 204 corresponds to an image 122 of a portion of the first item 204A which includes identifiable information (e.g., text, color, logos, patterns, pictures, images etc.) which is unique to the first item 204A and/or otherwise may be used to identify the first item 204A. A back image 122 of the first item 204A corresponds to an image 122 of a portion of the first item 204 which does not include identifiable information that can be used to identify the first item 204A. The machine learning model may be trained using a data set including known front images 122 and back images of items 204 of the first item 204A identified in the encoded vector library 128. Once each cropped image 2702 of the unidentified first item 204A is identified (e.g., tagged) as a front image 122 or a back image 122 of the first item 204A, item tracking device 104 discards all cropped images 2702 that were identified as back images 122. Item tracking device 104 selects an item identifier 1604 for the unidentified first item 204A from only those item identifiers 1604 corresponding to cropped images 2702 identified as front images 122 of the first item 204A. In a particular embodiment, after discarding all cropped images 2702 of the first item 204A that were identified as back images 122, if only one cropped image 2702 remains that was identified as a front image 122 of the first item 204A, item tracking device 104 selects the item identifier 1604 corresponding to the one remaining cropped image 2702. In case all cropped images 2702 of the first item 204A were identified as back images 122, the item tracking device 104 displays the item identifiers 1604 corresponding to one or more cropped images 2702 of the first item 204A on a user interface device and asks the user to select one of the displayed item identifiers 1604. Alternatively, item tracking device 104 may display instructions on the user interface device for the user to flip or rotate the first item 204A on the platform 202. Once the first item 204A has been flipped or rotated on the platform 202, item tracking device 104 may perform operations 2902-2906 to re-identify the first item 204A.

[0286] In some cases, multiple cropped images 2702 of the first item 204A may be identified as front images 122. In such cases, item tracking device 104 may be configured to select an item identifier 1604 from the item identifiers 1604 corresponding to cropped front images 122 of the item 204, based on the similarity values 1710 used to identify the respective item identifiers 1604 from the encoded vector library 128. As described above, for each cropped image 2702, item tracking device 104 selects an entry 1602 from the encode vector library 128 that is associated with the highest numerical similarity value 1710 in the similarity vector 1704 generated for the cropped image 2702. Item tracking device 104 then identifies the item identifier 1604 from the encoded vector library 128 that is associated with the identified entry 1602. Thus, the item identifier 1604 identified for each cropped image 2702 of the first item 204A corresponds to a respective similarity value 1710 based on which the item identifier 1604 was selected from the encoded vector library 128.

[0287] In one embodiment, among the cropped front images 2702 of the first item 204A, item tracking device 104 discards all cropped front images 2702 whose item identifiers 1604 were selected from the encoded vector library 128 based on numerical similarity values 1710 that are below a threshold similarity value. Since a similarity value 1710 is indicative of a degree of similarity between the encoded vector 1702 generated for an unidentified first item 204A and a particular encoded vector 1606 from the encoded vector library 128, a lower similarity value 1710 indicates a lower similarity between the generated encoded vector 1702 and corresponding encoded vector 1606 from the encoded vector library 128. By discarding all cropped front images 2702 whose item identifiers 1604 were selected from the encoded vector library 128 based on numerical similarity values 1710 that are below the threshold similarity value, item tracking device 104 discards all those cropped images 2702 that are unlikely to correctly identify the unidentified first item 204A. In an embodiment, if item identifiers 1604 of all cropped front images 2702 of the item 204 were selected from the encoded vector library 128 based on numerical similarity values 1710 that are below a threshold similarity value, item tracking device 104 displays the item identifiers 1604 on the user interface device and asks the user to select one of the displayed item identifiers 1604.

[0288] After discarding all cropped front images 2702 whose item identifiers 1604 were selected from the encoded vector library 128 based on numerical similarity values 1710 that are below the threshold value, item tracking device 104 applies a majority voting rule to select an item identifier 1604 from the item identifiers 1604 corresponding to the remaining cropped front images 2702 whose item identifiers 1604 were selected from the encoded vector library 128 based on numerical similarity values 1710 that equal or exceed the threshold similarity value. The majority voting rule defines that when a same item identifier 1604 has been identified for a majority of the remaining cropped front images 2702 of the unidentified item 204, the same item identifier 1604 is to be selected.

[0289] However, when no majority exists among the item identifiers 1604 of the remaining cropped front images, the majority voting rule cannot be applied. For example, when a same item identifier 1604 was not identified for a majority of the remaining cropped front images 2702 of the unidentified first item 204, the majority voting rule does not apply. In such cases, item tracking device 104 compares the two highest numerical similarity values 1710 among the remaining cropped front images 2702. When the difference between the highest similarity value and the second highest similarity value equals or exceeds a threshold difference, item tracking device 104 selects an item identifier 1604 that corresponds to the highest similarity value. However, when the difference between the highest similarity value and the second highest similarity value is below the threshold difference, item tracking device 104 displays the item identifiers 1604 corresponding to one or more remaining cropped front images 2702 of the first item 204A on the user interface device and asks the user to select one of the displayed item identifiers 1604.

[0290] Regardless of the particular method used to identify the first item 204A, an end result of this entire process is that a first item identifier 1604a is identified for the first item 204A.

[0291] At operation 2908, item tracking device 104 assigns the first item identifier 1604a to the first item 204A captured in the first images 122A.

[0292] At operation 2910, item tracking device 104 detects a second triggering event at the platform 202, wherein the second triggering event corresponds to the placement of a second item 204B on the platform 202. In a particular embodiment, the second triggering event may correspond to a user placing the second item 204B on the platform 202. Item tracking device 104 may detect the second triggering event similar to detecting the first triggering event described above with reference to operation 2902.

[0293] At operation 2912, in response to detecting the second triggering event, item tracking device 104 captures a plurality of second images 122B (e.g., as shown in FIG. 5B) of the second item 204B using two or more cameras 108 of the plurality of cameras 108.

[0294] At operation 2914, item tracking device 104 generates a plurality of cropped images 2704 (as shown in FIG. 27), wherein each cropped image (e.g., 2704a, 2704b, 2704c and 2704d) is associated with a corresponding second image 122b and is generated by editing the corresponding second image 122B to isolate at least a portion of the second item 204B.

[0295] To generate the plurality of cropped images 2704 of the second item 204B, item tracking device 104 may use a method similar to the method described above with reference to operation 2906 for generating cropped images 2702 of the first item 204A based on the first images 122A.

[0296] At operation 2916, for each cropped image 2704 of the second item 204B generated from the respective second image 122B, item tracking device 104 identifies an item identifier 1604 based on the attributes of the second item 204B in the cropped image 122B.

[0297] Item tracking device 104 may identify an item identifier 1604 for each cropped image 2704 of the second item 204B based on a method similar to the method described above with reference to operation 2906 for identifying an item identifier 1604 for each cropped image 2702 of the first item 204A.

[0298] At operation 2918, item tracking device 104 accesses (e.g., from the memory 116) associations 2802 between item identifiers 1604 of respective items 204.

[0299] At operation 2920, based on the associations 2802 stored in the memory 116, item tracking device 104 identifies an association 2802a between the first item identifier 1604a identified for the first item 204A and a second item identifier 1604b. Based on searching the associations 2802 stored in the memory 116, item tracking device 104 may determine that an association 2802a (e.g., association-2) exists between the first item identifier 1604a from entry 1602b and a second item identifier 1604b from entry 1602c. Following the example described above, the first item identifier 1604a from entry 1602b may be associated with a 1-liter bottle of soda and the second item identifier 1604b from entry 1602c may be associated with a small bag of chips.

[0300] At operation 2922, item tracking device 104 checks whether at least one of the item identifiers 1604, among item identifiers 1604 identified for the cropped images 2704 of the second item 204B, is the second item identifier 1604b. If none of the item identifiers 1604 identified for the cropped images 2704 of the second item 204B is the second item identifier 1604b, method 2800 proceeds to operation 2924 where item tracking device 104 displays the item identifiers 1604 of the cropped images 2704 of the second item 204B on the user interface device and asks the user to select one of the displayed item identifiers 1604.

[0301] However, if at least one of the item identifiers 1604 among item identifiers 1604 identified for the cropped images 2704 of the second item 204B is the second item identifier 1604b, the method 2900 proceeds to operation 2926 where item tracking device 104 assigns the second item identifier 1604b to the second item 204B captured in the second images 122B. Following the example described above, when the first item 204A is assigned the first item identifier 1604a from entry 1602b associated with a 1-liter bottle of soda, and at least one of the item identifiers 1604 among item identifiers 1604 identified for the cropped images 2704 of the second item 204B is the second item identifier 1604b from entry 1602c associated with a small bag of chips, item tracking device 104 assigns the second item identifier 1604b from entry 1602c to the second item 204B, thus identifying the second item 204 as a small bag of chips.

[0302] Following a second example of association-1 described above, when the first item 204A is assigned the first item identifier 1604 (I1) from entry 1602a associated with a 16 oz water bottle, and at least one of the item identifiers 1604 among item identifiers 1604 identified for the cropped images 2704 of the second item 204B is also the first item identifier 1604 (I1) from entry 1602a, item tracking device 104 assigns the same first item identifier 1604 (I1) from entry 1602a to the second item 204B as well. In this example, the first item identifier 1604 and the second item identifier 1604 are two different instances of the same item identifier 1604 (I1) from entry 1602a, and the first item 204A and the second item 204B are two different instances of the same item 204, for example two different 16 oz water bottles.

[0303] In one or more embodiments, item tracking device 104 applies the associations 2802 based logic described above to identify the second item 204B when one or more other methods described above for identifying the first item 204A do not apply or otherwise fail to identify the second item 204B.

[0304] In one embodiment, after generating cropped images 2704 for each second image 122B of the unidentified second item 204B, item tracking device 104 inputs each cropped image 2704 of the second item 204 into a machine learning model which determines whether the cropped image 2702 of the second item 204B is a front image 122 of the second item 204B or a back image 122 of the item second item 204B. Once each cropped image 2704 of the second item 204B is identified as a front image 122 or a back image 122 of the second item 204B, item tracking device 104 discards all cropped images 2704 that were identified as back images 122. Item tracking device 104 selects an item identifier 1604 for the unidentified second item 204B from only those item identifiers 1604 corresponding to cropped images 2704 identified as front images 122. For example, after discarding all cropped images 2704 of the second item 204 that were identified as back images 122, if only one cropped image 2704 remains that was identified as a front image 122, item tracking device 104 selects the item identifier 1604 corresponding to the one remaining cropped image 2704. In case all cropped images 2704 of the second item 204B were identified as back images 122, the item tracking device 104 displays the item identifiers 1604 corresponding to one or more cropped images 2704 on a user interface device and asks the user to select one of the displayed item identifiers 1604. Alternatively, item tracking device 104 may display instructions on the user interface device for the user to flip or rotate the second item 204B on the platform 202. Once the second item 204B has been flipped or rotated on the platform 202, item tracking device 104 may perform operations 2910-2916 to re-identify the second item 204b.

[0305] When multiple cropped images 2704 of the second item 204B are identified as front images 122, item tracking device 104 selects an item identifier 1604 from the item identifiers 1604 corresponding to cropped front images 2704 of the second item 204B, based on the similarity values 1710 used to identify the respective item identifiers 1604 from the encoded vector library 128. As described above with reference to cropped images 2702 of the first item 204B, for each cropped image 2704 of the second item 204B, item tracking device 104 selects an entry 1602 from the encoded vector library 128 that is associated with the highest numerical similarity value 1710 in the similarity vector 1704 generated for the cropped image 2704. Item tracking device 104 then identifies the item identifier 1604 from the encoded vector library 128 that is associated with the identified entry 1602. Thus, the item identifier 1604 identified for each cropped image 2704 of the second item 204B corresponds to a respective similarity value 1710 based on which the item identifier 1604 was selected from the encoded vector library 128.

[0306] In one embodiment, among the cropped front images 2704 of the second item 204B, item tracking device 104 discards all cropped front images 2704 whose item identifiers 1604 were selected from the encoded vector library 128 based on numerical similarity values 1710 that are below a threshold similarity value. Since a similarity value 1710 is indicative of a degree of similarity between the encoded vector 1702 generated for the unidentified second item 204B and a particular encoded vector 1606 from the encoded vector library 128, a lower similarity value 1710 indicates a lower similarity between the generated encoded vector 1702 and corresponding encoded vector 1606 from the encoded vector library 128. By discarding all cropped front images 2704 whose item identifiers 1604 were selected from the encoded vector library 128 based on numerical similarity values 1710 that are below the threshold value, item tracking device 104 discards all those cropped images 2704 that are unlikely to correctly identify the unidentified second item 204B. In an embodiment, if item identifiers 1604 of all cropped front images 2704 of the second item 204B were selected from the encoded vector library 128 based on numerical similarity values 1710 that are below the threshold similarity value, item tracking device 104 displays the item identifiers 1604 on the user interface device and asks the user to select one of the displayed item identifiers 1604.

[0307] After discarding all cropped front images 2704 whose item identifiers 1604 were selected from the encoded vector library 128 based on numerical similarity values 1710 that are below the threshold similarity value, item tracking device 104 applies a majority voting rule to select an item identifier 1604 from the item identifiers 1604 corresponding to the remaining cropped front images 2704 whose item identifiers 1604 were selected from the encoded vector library 128 based on numerical similarity values 1710 that equal or exceed the threshold similarity value. The majority voting rule defines that when a same item identifier 1604 has been identified for a majority of the remaining cropped front images 2704 of the unidentified second item 204B, the same item identifier 1604 is to be selected.

[0308] However, when no majority exists among the item identifiers 1604 of the remaining cropped front images 2704, the majority voting rule cannot be applied. For example, when a same item identifier 1604 was not identified for a majority of the remaining cropped front images 2704 of the unidentified second item 204B, the majority voting rule does not apply. In such cases, item tracking device 104, compares the two highest numerical similarity values 1710 among the remaining cropped front images 2704. When the difference between the highest similarity value and the second highest similarity value equals or exceeds a threshold difference, item tracking device 104 selects an item identifier 1604 that corresponds to the highest similarity value.

[0309] However, when the difference between the highest similarity value and the second highest similarity value is below the threshold difference, item tracking device 104 applies the associations-based logic described above with reference to operations 2918-2926.

Example System for Camera Re-Calibration Based on an Updated Homography

[0310] In general, certain embodiments of the present disclosure describe techniques for camera re-calibration based on an updated homography in response to a shift in position of any system component, such as a camera, 3D sensor or a platform. For example, the disclosed system is configured to detect if there is shift in position of any of the camera, 3D sensor, and/or platform, and in response to detecting the shift in position of any of the camera, 3D sensor, and platform, generate a new homography, and re-calibrate the camera and 3D sensor using the new homography. In this manner, the disclosed system improves the item identifying and tracking techniques. For example, the disclosed system increases the accuracy in item tracking and identification techniques, specifically, in cases where a camera, a 3D sensor, and/or platform has moved from its initial position when the initial homography was generated and determined. Accordingly, the disclosed system provides the practical application and technical improvements to the item identification and tracking techniques. For example, the disclosed system offers technical improvements in the field of item identification and tracking technology by addressing the inherent challenge of maintaining accuracy in a dynamic environment. For example, the disclosed system may continuously or periodically (e.g., every second, every milliseconds, etc.) monitor the positions of cameras, 3D sensors, and the platform. When the disclosed system detects any shift in the location of any of these components, the disclosed system generates a new homography and recalibrates the cameras and 3D sensors accordingly. Therefore, the pixel-to-physical location mapping remains precise (or within an acceptable precision threshold), even in scenarios where one or more system components (e.g., cameras, 3D sensors, and platform) have been moved or shifted. Furthermore, the disclosed system increases reliability by proactively addressing challenges of shifts in locations of cameras, 3D sensors, and the platform and maintains high accuracy even in changing conditions. In this manner, the disclosed system provides additional practical applications and technical improvements to the item identification and tracking technology. Accordingly, this represents an improvement to the efficiency, throughput, and productivity of computer systems implemented to perform the described operations.

[0311] FIG. 30 illustrates an embodiment of a system 3000 configured to detect if there is any change in the initial homography 608 used to translate between pixel locations in an image 122, 124 captured by a camera 108 or 3D sensor 110 and physical locations in a global plane; generate an updated homography 3038; and calibrate or re-calibrate camera 108 and/or 3D sensor 110 based on the updated homography 3038. FIG. 30 further illustrates an example operational flow 3050 of the system 3000 for camera calibration/re-calibration based on the updated homography 3038. In some embodiments, the system 3000 includes the item tracking device 104 communicatively coupled with the imaging device 102, via a network 106. In the example of FIG. 30, the configuration of imaging device 102 described in FIG. 2A is used. However, the configuration of imaging device 102 described in FIG. 2B or any other configuration of the imaging device 102 may be used in the system 3000. In the example configuration of imaging device 102 in FIG. 30, the imaging device 102 includes cameras 108a-d, 3D sensor 110, structure 206, weight sensor 112, and platform 202. In some configurations of the imaging device 102, any number of cameras 108, 3D sensors 110, and weight sensors 112 may be implemented, similar to that described in FIGS. 1, 2A, and 2B. The 3D sensor 110 may also interchangeably be referred to as a 3D camera or camera. The system 3000 may be configured as shown in FIG. 30 or in any other configuration. The systems and components illustrated and described in the discussions of any of the figures may be used and implemented to perform operations of the systems and methods described in FIGS. 30-31. Additionally, systems and components illustrated and described with reference to any figure of this disclosure may be used and implemented to perform operations of the systems and methods described in FIGS. 30-31.

[0312] In general, the system 3000 increases the accuracy in item tracking and identification techniques, specifically, in cases where a camera 108, a 3D sensor 110, and/or platform 202 has moved from its initial position when the homography 608 was generated and determined. The process of generating the homography 608 is described above in conjunction with the discussion of FIG. 11 and the process of using the homography 608 to determine a physical location of an item from pixel locations where the item is shown in an image captured by a camera 108 is described in conjunction with FIGS. 12A-B.

[0313] In some embodiments, when the cameras 108, 3D sensor 110, and the platform 202 are deployed onto the imaging device 102, the cameras 108 and 3D sensor 110 may be calibrated during an initial calibration so pixel locations in an image 122, 124 captured by a given camera 108 and 3D sensor 110 are mapped to respective physical locations on the platform 202 in the global plane. For example, during the initial calibration of cameras 108, a paper printed with unique patterns of checkboards may be placed on the platform 202. Each camera 108 may capture an image 122 of the paper and transmit to the item tracking device 104. The item tracking device 104 may generate the homography 608 that maps pixel locations of each unique pattern on the paper shown in the image 122 to corresponding physical locations of the unique pattern on the paper on the platform 202. Similar operations may be performed with respect to depth images 124 captured by the 3D sensor 110. For example, during the initial calibration of 3D sensor 110, the paper printed with unique patterns of checkboards may be placed on the platform 202. Each 3D sensor 110 may capture a depth image 124 of the paper and transmit it to the item tracking device 104. The item tracking device 104 may generate the homography 608 that maps pixel locations of each unique pattern on the paper shown in the image 124 to corresponding physical locations of the unique pattern on the paper on the platform 202.

[0314] After the initial calibration, the item tracking engine 114 may determine the physical location of any item (204 in FIG. 2A) placed on the platform 202 by applying the homography 608 to the pixel locations of the item shown in an image 122 or depth image 124. In some cases, a camera 108, 3D sensor 110, and/or the platform 202 may move or be shifted from its initial location due to any number of reasons, such as an impact from a person, movement by an item when it is being placed on the platform 202, and the like. Because the homography 608 is determined based on the locations of the camera 108, 3D sensor 110, and the platform 202, a change in the initial location of one or more of the camera 108, 3D sensor 110, and/or the platform 202 may lead to the homography 608 not being accurate anymore. As a result, applying the homography 608 to subsequent pixel locations of items shown in images 122 or depth images 124 may not lead to the actual physical location of the items on the platform 202, and vice versa—meaning that applying an inverse of the homography 608 to physical locations of subsequent items placed on the platform 202 may not lead to the actual pixel locations of the items shown in a respective image 122 or depth image 124.

[0315] In practice, it is very difficult, if not impossible, to know if a camera 108, 3D sensor 110, and/or the platform 202 is shifted in position if no one witnessed it or if it is not captured on a camera facing the imaging device 102. One potential solution to this problem of a camera 108, 3D sensor 110 and/or platform 202 being shifted, resulting in an inaccurate homography 608, is to provide routine maintenance to the cameras 108, 3D sensor 110, and platform 202 to ensure that they are not shifted from their respective original locations. However, this potential solution is not feasible given that the imaging device 102 may be deployed in a store and routine maintenance of the cameras 108, 3D sensor 110, and platform 202 will interrupt the item check-out process. Besides, routine maintenance is labor-intensive and requires precise measurement of the locations of the cameras 108, 3D sensor 110, and platform 202, which makes it an error-prone process.

[0316] The present disclosure provides a solution to this and other technical problems that are currently arising in the realm of item identification and tracking technology. For example, the system 3000 is configured to detect if there is a shift in location of camera 108, 3D sensor 110, and/or platform 202, and in response to detecting the shift in location of either camera 108, 3D sensor 110, and platform 202, generate a new homography 3038, and re-calibrate the camera 108, 3D sensor 110 using the new homography 3038. In this manner, the system 3000 improves the item identifying and tracking techniques.

System Components

[0317] Aspects of the item tracking device 104 are described in FIGS. 1-29, and additional aspects are described below. The item tracking device 104 may include the processor 602 in signal communication with the network interface 604 and the memory 115. The memory 116 stores software instructions 3002 that when executed by the processor 602 cause the processor 602 to execute the item tracking engine 114 to perform one or more operations of the item tracking device 104 described herein.

[0318] Memory 116 also stores homographies 608, 3038, machine learning model 126, pixel location array 3022, calculated global location array 3030, reference location array 3034, threshold value 3036, line detection algorithm 3042, edge detection algorithm 3040, and/or any other data or instructions. The software instructions 3002 may comprise any suitable set of instructions, logic, rules, or code operable to execute the processor 602 and item tracking engine 114 and perform the functions described herein. Machine learning model 126 is described with respect to FIGS. 1-6. Other elements are described further below in conjunction with the operational flow 3050 of the system 3000.

Calibration Board

[0319] The calibration board 3010 may include a pattern of lines embedded on the surface of the platform 202. In certain embodiments, instead of using a unique checkboard pattern that was used in the initial calibration of the cameras 108, 3D sensor 110, and generation of the homography 608, the calibration board 3010 that includes repeating patterns of lines and edges may be used for the camera re-calibration process and generating the new homography 3038. In one example, this may be because, for the initial camera calibration process and homography generation, the uniqueness of each checkboard helps to differentiate the different checkboard patterns, and therefore different locations of the respective checkboard patterns may be determined with a greater confidence score. In another example, embedding the checkboard into platform 202 when the imaging device 102 is deployed in a physical store may appear intimidating to users and may not be user-friendly.

[0320] In the camera re-calibration process and generation of the new homography 3038, the repeating patterns on the calibration board 3010 may suffice and a unique checkboard may not be needed. However, in some embodiments, the checkboard that was used in the camera calibration process may be used in the camera re-calibration process.

[0321] The lines and edges on the calibration board 3010 may be detected by the item tracking engine 114. For example, a camera 108 may capture an image 122 of the calibration board 3010 and transmit it to the item tracking device 104. In a case where the image 122 is a color image, the item tracking engine 114 may convert the image 122 into a gray-scale image. In some embodiments, to generate the image of the calibration board 3010, the item tracking engine 114 may determine the intersections between the lines shown in the image 122. To this end, the item tracking engine 114 may perform one or more of the operations below. The item tracking engine 114 may implement an edge detection algorithm 3040, such as Sobel filters, and the like, to detect edges of the pattern displayed on the image of the calibration board 3010. The edge detection algorithm 3040 may be implemented by the processor 602 executing the software instructions 3002 and is generally configured to detect edges and shapes in an image. The edge detection algorithm 3040 may include an image processing algorithm, neural network, and the like.

[0322] The item tracking engine 114 may implement a line detection algorithm 3042, such as, for example, a Hough transform, and the like, to detect the lines of the pattern displayed on the image of the calibration board 3010. The line detection algorithm 3042 may be implemented by the processor 602 executing the software instructions 3002 and is generally configured to detect lines in an image. The edge detection algorithm 3040 may include an image processing algorithm, neural network, and the like. In this process, the item tracking engine 114 feeds the image of the calibration board 3010 to the edge detection algorithm 3040 and line detection algorithm 3042 to generate the image 3012. In this process, the item tracking engine 114 may apply line and edge fitters and algorithms on the image 3012 to generate the image 3014 of the calibration board. The item tracking engine 114 may implement a noise reduction algorithm to reduce (or remove) noises on the pattern displayed on the image 3014 of the calibration board 3010, where the noises are small dots scattered around the lines and edges displayed on the image of the calibration board 3010. For example, if a dot is less than a threshold area (e.g., less than 5 centimeters (cm), 3 cm, etc.), the item tracking engine 114 may remove the dot from the image of the calibration board 3010. In this manner, the item tracking engine 114 may detect and draw the lines and intersections between the lines on the image 3014 of the calibration board 3010. Similar operations may be performed with respect to a color image 122 and/or depth image 124 of the calibration board 3010.

Operational Flow for Camera Re-Calibration Based on an Updated Homography

[0323] The operational flow 3050 of the system 3000 may begin when a camera 108 captures an image 122 and/or a 3D sensor 110 captures a depth image 124 and transmits the image 122 and/or the dept image 124 to the item tracking device 104. For simplicity and brevity, 3D sensor 110 may be referred to as a camera 108. For example, if the camera 108 is configured to capture color images 122 and depth images 124, the operations with respect to the camera 108 for camera re-calibration may also apply to the 3D sensor 110. The image 122 and/or the depth image 124 may display at least a portion of the platform 202. The image 122, 124 may be processed to detect lines and edges and the image 3014 of the calibration board 3010 may be produced by the item tracking engine 114, similar to that described above. The image 3014 of the calibration board 3010 may display or show points 3016-n that are intersecting points between each pair of lines on the image 3014. For clarity, the operations below are described with respect to the image 122. However, similar operations may be performed with respect to depth image 124.

[0324] The item tracking engine 144 may determine the pixel location array 3022 that comprises a set of pixel locations 3024a through 3024n, where each row in the pixel location array 3022 represents a pixel location of a single point 3016 that is an intersection of a pair of lines on the image 3014 of the calibration board 3010. For example, the first row as indicated by pixel location (PL) 11 represents the pixel location of a first point 3016a, and the n-th row as indicated by PL In represents the pixel location of the n-th point 3016n. In some embodiments, the item tracking engine 144 may determine the pixel location array 3022 by feeding the image 3014 to the machine learning model 126 which includes an image processing algorithm, similar to that described in FIGS. 1-3, and 7-26. Each of the pixel location array 3022, calculated location array 3030, homography 608, reference location array 3034, and second homography 3038 may have any dimension that leads to the correct matrix/array multiplication.

Detecting Physical Locations of Points on the Calibration Board by Using Initial Homography

[0325] The item tracking engine 144 may apply the first (initial) homography 608 to each pixel location 3024a-n to determine the respective physical location of each point 3016 in the global plane. In this process, the item tracking engine 144 may perform a matrix multiplication between the pixel location array 3022 and the first homography 608 if they are presented as matrices. In certain embodiments, each row of the pixel location array 3022 may be multiplied by the first homography 608 to determine the respective physical location of the point 3016 associated with the respective row of the pixel location array 3022. In some embodiments, the coefficients of the first homography 608 (see, e.g., FIGS. 12A-B) may be repeated in each column of the first homography 608 matrix so multiplication of each row of the pixel location array 3022 with a column of the first homography 608 matrix results in a respective calculated physical location coordinates associated with the respective row in the pixel location array 3022. By applying the first homography 608 to the pixel location array 3022, the item tracking engine 114 determines the calculated location array 3030.

[0326] The calculated location array 3030 identifies a set of calculated physical (x,y) location coordinates of the set of points 3016 on the calibration board in the global plane. Each row in the calculated location array 3030 represents a calculated physical location coordinate (x,y) of the respective point 3016 on the calibration board 3010 in the global plane. For example, the first row as indicated by global location (GL) 11 represents the calculated physical (x1,y1) location coordinates of the PL 11 of the point 3016a, and the n-th row as indicated by GL In represents the calculated physical (xn,yn) location coordinates of the PL In of the point 3016n. The item tracking engine 114 may repeat similar operations for each image 122, 124 that is captured by a respective camera 108, where for each camera 108, a different homography 608 is used because the coefficients of the homography 608 depend on the locations of the camera 108 with respect to the platform 202 in the global plane.

[0327] The item tracking engine 114 may perform similar operations to determine the calculated physical locations of items that are placed on the platform 202 with respect to the previously determined and known physical locations of the points 3016 on platform 202 for each captured image 122, 124 captured by each camera 108, similar to that described in FIGS. 1-29.

Detecting Whether the Camera and/or the Platform is Moved

[0328] In cases where the camera 108, 3D sensor 110, and/or the platform 202 has moved (or shifted in position), the first homography 608 may not be accurate, and therefore, the calculated physical location coordinates (x,y) of points 3016 may not be the actual physical location coordinates (x,y) of the points 3016. To determine if the camera 108, 3D sensor 110, and/or the platform 202 has moved, the item tracking engine 114 may compare the newly calculated location array 3030 with the reference location array 3034. The reference location array 3034 may be the ground truth that represents the actual and previously verified physical location coordinates (x,y) of the points 3016 in the global plane.

[0329] The reference location array 3034 may be determined by applying the first homography 608 on to the pixel location array 3022 under known conditions where the cameras 108 and platform 202 are at their respective expected locations. For example, a first row of the reference location array 3034 represented by GL′ 11 may represent the actual physical location coordinates (x1′,y1′) of the first point 3016a, and the n-th row of the reference location array 3034 represented by GL′ In may represent the actual physical location coordinates (xn′, yn′) of the n-th point 3016n on the calibration board 3010. By comparing the newly calculated location array 3030 with the reference location array 3034, the item tracking engine 114 may determine whether there is any change or shift in the calculated physical locations of the points 3016 as a result of a change in the location of the camera 108, 3D sensor 110, and/or the platform 202. In this process, the item tracking engine 114 may perform a vector comparison between the newly calculated location array 3030 and the reference location array 3034.

[0330] The item tracking engine 114 may determine the difference between the newly calculated location array 3030 and the reference location array 3034. The item tracking engine 114 may determine whether the determined difference between the newly calculated location array 3030 and the reference location array 3034 is more than a threshold value 3036. The threshold value 3036 may be 3%, 4%, and the like. For example, in some embodiments, the item tracking engine 114 may determine a Euclidean distance between the newly calculated location array 3030 and the reference location array 3034. If the determined Euclidean distance is more than a threshold distance value (e.g., more than 0.1 cm, 0.3 cm, etc.), it may be determined that the difference between the newly calculated location array 3030 and the reference location array 3034 is more than the threshold value 3036. Otherwise, the item tracking engine 114 may determine that the difference between the newly calculated location array 3030 and the reference location array 3034 is less than the threshold value 3036. In other examples, the item tracking engine 114 may use any other type of distance calculations between the newly calculated location array 3030 and the reference location array 3034.

[0331] In another example, in some embodiments, the item tracking engine 114 may determine the difference between the reference location array 3034 and the calculated location array 3030 by performing a dot product operation between each element of the reference location array 3034 and the respective element in the calculated location array 3030. The dot product is a multiplication of the absolute values of the two elements multiplied by a cosine of an angle between the two elements, where each of the two elements is a vector representing a location coordinate of the respective point 3016.

[0332] In another example, in some embodiments, the item tracking engine 114 may determine the difference between the reference location array 3034 and the calculated location array 3030 by performing a dot product operation between the newly calculated location array 3030 and the reference location array 3034.

Generating a Second Homography

[0333] If the item tracking engine 114 determines that the difference between the newly calculated location array 3030 and the reference location array 3034 is more than the threshold value 3036, it may be an indication that the camera 108, 3D sensor 110, and/or the platform 202 has moved from its respective initial location when the first homography 608 was generated and determined that resulted in a significant difference or shift in the newly calculated physical locations coordinates (x,y) of the points 3016 in the calculated location array 3030 compared to the reference location array 3034. In response, the item tracking engine 114 may determine that the camera 108 and/or the platform 202 has moved from their respective initial location when the first homography 608 was determined and used to determine the reference location array 3034.

[0334] In response, the item tracking engine 114 may determine a second homography 3038. In this process, the item tracking engine 114 may determine the inverse of the physical location array 3022 and compute the multiplication of the inverse of the physical location array 3022 with the reference location array 3034. Similarly, if matrix format is used, the inverse matrix of the physical location matrix maybe multiplied with the reference location matrix. The result of this multiplication is the second homography 3038 that is configured to translate between the pixel locations 3024a-n of the points 3016 in an image of the calibration board 3010 and the reference location array 3034.

[0335] The item tracking engine 114 may then use the second homography 3038 to re-calibrate or calibrate the camera 108. In this process, the item tracking engine 114 may use the second homography 3038 in determining the physical locations of items that are placed on the platform 202, similar to that described in FIGS. 1-29. In other words, the item tracking device 104 may use the second homography 3038 to translate between a pixel location in an image 122, 124 and a respective physical location coordinate (x,y) in the global plane. For example, the item tracking device 104 may project from (x,y) coordinates indicated in the reference location array 3034 in the global plane to pixel locations in the pixel location array 3022 associated with an image 122 or depth image 124. For example, the item tracking device 104 may receive an (x,y) physical coordinate (e.g., GL′ 11) for an object in the global plane. The item tracking device 104 identifies a homography 3038 that is associated with a camera 108 or 3D sensor 110 where the object is seen. The item tracking device 104 may then apply the inverse homography 3038 to the (x,y) physical coordinate to determine a pixel location 3024a where the object is located in the image 122 or depth image 124. To this end, the item tracking device 104 may compute the matrix inverse of the homography 3038 when the homography 3038 is represented as a matrix. For example, the item tracking device 104 may perform matrix multiplication between an (x,y) coordinates (e.g., GL′ 11) of an object in the global plane and the inverse homography 3038 to determine a corresponding pixel location 3024a of the object (e.g., PL 11) in the image 122 or depth image 124.

[0336] In another example, the item tracking device 104 may receive a pixel location of an object in an image 122 or depth image 124. The item tracking device 104 may identify the homography 3038 that is associated with a camera 108 or 3D sensor 110 that captured the image 122 or depth image 124. The item tracking device 104 may then apply homography 3038 to the pixel location of the object to determine the physical location of the object in the global plane. For example, the item tracking device 104 may perform matrix multiplication between the pixel location 3024a of the object in an image 122 or a depth image 124 and the homography 3038 to determine the physical location coordinate (x,y) of the object in the global plane. In this manner, the item tracking device 104 may translate between pixel locations in an image 122 or a depth image 124 and respective physical locations using the homography 3038.

[0337] In certain embodiments, the second homography 3038 may comprise coefficients that translate between the pixel location array 3022 in the image 122, 124 and the reference location array 3034 in the global plane.

[0338] The item tracking engine 114 may perform similar operations with respect to each camera 108 that a change more than the threshold value 3036 is detected, similar to that described above. For example, with respect to a second camera 108, the item tracking engine 114 may receive an image 122, 124 from the second camera 108, where the image 122, 124 shows at least a portion of the set of points 3016 on the calibration board 3010. The image 122, 124 may include the image 3014. In this example, the second camera 108 may be at a different location than the first camera 108 described above. Therefore, the second image 122, 124 may be captured at a different angle compared to the first image 122, 124 described above.

[0339] The item tracking engine 114 may determine a second pixel location array 3022 that comprises a second set of pixel locations 3024a-n associated with the set of points 3016 in the second image 122, 124. The item tracking engine 114 may determine a second calculated location array 3030 by applying the first homography 608 to the second pixel location array 3022. The second calculated location array 3030 identifies a second set of calculated physical (x,y) location coordinates of the set points 3016 in the global plane. The item tracking engine 114 may compare the reference location array 3034 with the second calculated location array 3030.

[0340] The item tracking engine 114 may determine whether there is a difference between the reference location array 3034 and the second calculated location array 3030 and if the difference is more than the threshold value 3036. If it is determined that the difference between the reference location array 3034 and the second calculated location array 3030 is more than the threshold value 3036, the item tracking engine 114 may determine that the second camera 108 and/or the platform 202 has moved from its respective location from when the first homography 608 was determined. In response, the item tracking engine 114 may determine a third homography 3038 by multiplying an inverse of the second pixel location array 3022 and the reference location array 3034. The item tracking engine 114 may then calibrate the second camera 108 using the third homography 3038 to determine the physical locations of items that are placed on the platform 202.

[0341] In certain embodiments, comparing the reference location array 3034 with the calculated location array 3030 may include comparing each element in the reference location array 3034 with a counterpart element in the calculated location array 3030. For example, GL 11 may be compared with GL′ 11, and GL In may be compared with GL′ In. The threshold value 3036 may be with respect to differences between each element in the calculated location array 3030 and the respective element in the reference location array 3034. For example, the threshold value 3036 may correspond to an accumulation or an average of differences between each element in the calculated location array 3030 and the respective element in the reference location array 3034.

[0342] In certain embodiments, to determine the pixel location array 3022, the item tracking engine 114 may convert the image 122, 124 into a gray-scale image 122, 124, remove noises, such as areas or dots that have less than a threshold circumference (e.g., less than 3 cm, 4 cm, etc.) from the gray-scale image 122, 124, detect a set of lines on the image 3014 of the calibration board 3010 (e.g., via a line detection algorithm), detect a set of intersections where each pair of lines meet on the image 3014 of the calibration board 3010 (e.g., via an edge detection algorithm), determine a pixel location 3024a-n of each intersection from among the set of intersections, and form the set of pixel locations 3024a-n of the set of intersections in the pixel location array 3022.

Example Method for Camera Re-Calibration Based on Updated Homography

[0343] FIG. 31 illustrates an example flow chart of a method 3100 for camera re-calibration based on an updated homography 3038. Modifications, additions, or omissions may be made to method 3100. Method 3100 may include more, fewer, or other operations. For example, operations may be performed in parallel or in any suitable order. While at times discussed as the system 3900, item tracking device 104, item tracking engine 114, imaging device 102, or components of any of thereof performing operations, any suitable system or components of the system may perform one or more operations of the method 3100. For example, one or more operations of method 3100 may be implemented, at least in part, in the form of software instructions 3002 of FIG. 30, stored on a tangible non-transitory computer-readable medium (e.g., memory 116 of FIG. 30) that when run by one or more processors (e.g., processors 602 of FIG. 30) may cause the one or more processors to perform operations 3102-3118.

[0344] At operation 3102, the item tracking engine 114 receives an image 122, 124 from a camera 108, where the image 122, 124 shows at least a portion of the set of points 3016 on the calibration board 3010. For example, the item tracking engine 114 may receive the image 122, 124 when the imaging device 102 sends the image 122, 124 to the item tracking device 104.

[0345] At operation 3104, the item tracking engine 114 determines the pixel location array 3022 that comprises a set of pixel locations 3024a-n associated with the set of points 3016 in the image 122, 124. For example, the item tracking engine 114 may feed the image 122, 124 to the image processing algorithm included in the machine learning module 126, similar to that described in FIGS. 1-29.

[0346] At operation 3106, the item tracking engine 114 determines, by applying a first homography 608 to the pixel location array 3022, the calculated location array 3030 that identifies calculated physical (x,y) location coordinates of the set of points 3016 in the global plane. For example, the item tracking engine 114 may multiply each element of the pixel location array 3022 with the first homography 608 to determine the respective calculated physical location coordinate (x,y) of the respective point 3016 in the global plane.

[0347] At operation, 3108, the item tracking engine 114 compares the reference location array 3034 with the calculated location array 3030. At operation 3110, the item tracking engine 114 determines a difference between the reference location array 3034 and the calculated location array 3030.

[0348] At operation 3112, the item tracking engine 114 determines whether the difference between the reference location array 3034 and the calculated location array 3030 is more than the threshold value 836. If it is determined that the difference between the reference location array 3034 and the calculated location array 3030 is more than the threshold value 836, method 3100 proceeds to operation 3114. Otherwise, method 3100 returns to operation 3102 to evaluate another image 122, 124 captured by the same or another camera 108.

[0349] At operation 3114, the item tracking engine 114 determines that the camera 108, (3D sensor 110), and/or the platform 202 have/has moved from a respective initial location when the first homography 608 was determined. At operation 3116, the item tracking engine 114 determines a second homography 3038 by multiplying an inverse of the pixel location array 3022 with the reference location array 3034. At operation 3118, the item tracking engine 114 calibrates the camera 108 and/or 3D sensor 110 using the second homography 3038.

Detecting a Trigger Event for Identification of an Item

[0350] In general, certain embodiments of the present disclosure describe techniques for detecting a triggering event corresponding to a placement of an item on a platform (e.g., platform 202 shown in FIG. 2A) of an imaging device (e.g., imaging device 102 shown in FIG. 2A). An overhead camera positioned above the platform and having a top view of the platform is configured to take pictures of the platform (e.g., periodically or continually). Each particular pixel of an image captured by the overhead camera is associated with a depth value indicative of a distance between the overhead camera and a surface depicted by the particular pixel. A reference image of an empty platform is captured and an average reference depth value associated with all pixels in the reference image is calculated. Thereafter, for each subsequent image captured by the overhead camera, a real-time average depth associated with all pixels of the subsequent image is calculated and subtracted from the reference depth calculated for the empty platform. When the difference between the reference depth and real-time depth stays constant above zero across several images of the platform, it means that an item has been placed on the platform and is ready for identification. In response, a triggering event is determined to have been detected.

[0351] Item tracking device 104 may be configured to detect a triggering event at the platform 202 of an imaging device 102 (shown in FIG. 2A), wherein the triggering event corresponds to the placement of an item 204 on the platform 202.

[0352] The item tracking device 104 detects a triggering event corresponding to the placement of an item 204 on the platform 202 of the imaging device 102 by detecting that a user's hand holding the first item 204A entered the platform 202, placed the first item 204A on the platform 202, and exited the platform 202.

[0353] As described further below in accordance with certain embodiments of the present disclosure, the item tracking device 104 can use a depth image 124 (e.g., shown in FIG. 33A) of an empty platform 202 or a depth image 124 of the platform 202 with one or more items 204 already placed on the platform 202 as a reference overhead depth image 3302 (shown in FIG. 33A). At a later time, the item tracking device 104 can detect that an item 204 or an additional item 204 has been placed on the surface 208 of the platform 202 based on differences in depth values of pixels between subsequent depth images 124 (e.g., secondary depth images 3304 shown in FIGS. 33A, 33B and 33C) of the platform 202 and the reference depth image 3302.

[0354] For example, as shown in FIG. 33A, item tracking device 104 first captures a reference overhead depth image 3302, wherein the reference overhead depth image 3302 is captured by a 3D sensor 110 (shown in FIG. 2A) that is positioned above the platform 202 and that is configured to capture overhead depth images 124 of the platform 202. In one embodiment, the reference overhead depth image 3302 is of the platform 202 without any obstructions between the 3D sensor 110 and the surface 208 of the platform 202. Each pixel in the reference overhead depth image 3302 is associated with a depth value (d) 111 (shown in FIG. 2A) indicating a distance between the 3D sensor 110 and a portion of the surface of the platform 202 depicted by the pixel. Item tracking device 104 determines a reference average depth value across all pixels of the reference overhead depth image 3302. Subsequently, in response to detecting a motion by a proximity sensor 250 (shown in FIG. 2A) of the imaging device 102, item tracking device 104 starts capturing secondary overhead depth images 3304 (shown as 3304a in FIG. 33B, 3304b in FIG. 33C, and 3304c at FIG. 33D) of the platform 202 using the 3D sensor 110 (shown in FIG. 2A). Like the reference overhead depth image 3302, each pixel of each secondary overhead depth image 3304 is associated with a depth value indicating a distance (d) 111 (shown in FIG. 2A) between the 3D sensor 110 and a surface (e.g., surface of the platform 202, a hand or an item 204, etc.) depicted by the pixel. Item tracking device 104 compares each secondary overhead depth image 3304 with the reference overhead depth image 3302 to determine one or more events (shown as events E1, E2 and E3 in FIG. 34) associated with identifying a triggering event corresponding to placement of an item 204 on the platform 202. For example, item tracking device 104 calculates a depth difference parameter (D) 3402 (shown in FIG. 34) based on comparing each captured secondary overhead depth image 3304 captured by the 3D sensor 110 with the reference overhead depth image 3302. The depth difference parameter (D) 3402 is a single numerical value that represents a comparison of the depth values associated with pixels in the secondary overhead depth image 3304 and the pixels in the reference overhead depth image 3302.

[0355] FIG. 34 illustrates a plot 3400 of the depth difference parameter (D) 3402 over time (t). By tracking the value of the depth difference parameter (D) 3402 across a plurality of secondary overhead depth images 3304, item tracking device 104 determines whether a triggering event has occurred at the platform 202 of the imaging device 102. As shown in FIG. 34, item tracking device detects a first event E1 when D exceeds a pre-set threshold value (Th) at a secondary overhead depth image 3304 (e.g., 3304a shown in FIG. 33B) that is captured at time instant t1. Event E1 may indicate that a user's hand 3306 holding item 204 has entered the view of the 3D sensor 110 and is moving inward on the platform 202 as shown in FIG. 33B. Referring back to FIG. 34, item tracking device detects a second event E2 when the value of D starts dropping, indicating that the user's hand 3306 has placed the item 204 on the platform 202 and is moving away from the platform 202, as shown in FIG. 33C. Referring back to FIG. 34, item tracking device detects a third event E3 in response to detecting that the value of D has stayed constant at a value higher than a threshold for a given time interval, indicating that the user's hand 3306 has completely moved away from the platform 202 and is no more in view of the 3D sensor 110. This is shown in FIG. 33D where item 204 is placed on the platform 202 and the user's hand 3306 is not visible. In response to determining that the value of D has stayed constant at a value higher than the threshold for a given time interval, item tracking device 104 may determine that a triggering event has occurred at the platform 202 corresponding to placement of the item 204 on the platform 202. These aspects will now be described below in further detail with reference to FIGS. 32A, 32B, 33A-D and 34.

[0356] The system and method described in certain embodiments of the present disclosure provide a practical application of intelligently detecting a triggering event corresponding to placement of an item 204 on the platform 202 of the imaging device 102. As described with reference to FIGS. 32A-B, 33A-D, and 34, the item tracking device 104 detects whether an item 204 has been placed on the platform 202 by comparing a reference overhead image of an empty platform 202 with a plurality of subsequently captured overhead images of the platform 202. By calculating a difference in the average depth values associated with pixels of the reference image and the plurality of subsequent images, the item tracking device 104 determines, for example, that a user's hand holding an item 204 entered the platform 202, placed the first item 204 on the platform 202, and exited the platform 202. This technique for detecting a triggering event avoids false detection of triggering events as well as avoids missed detection of triggering events, thus improving accuracy associated with detecting triggering events at the platform 202. Further, by avoiding false detection of triggering events, the disclosed system and method saves computing resources (e.g., processing and memory resources associated with the item tracking device 104) which would otherwise be used to perform one or more processing steps that follow the detection of a triggering event such as capturing images using cameras 108 of the imaging device 102 to identify items 204 placed on the platform 202. This, for example, improves the processing efficiency associated with the processor 602 (shown in FIG. 6) of the item tracking device 104. Thus, the disclosed system and method generally improve the technology associated with automatic detection of items 204.

[0357] It may be noted that the systems and components illustrated and described in the discussions of FIGS. 1-29 may be used and implemented to perform operations of the systems and methods described in FIGS. 32A-B, 33A-D, and 34. Additionally, systems and components illustrated and described with reference to any figure of this disclosure may be used and implemented to perform operations of the systems and methods described in FIGS. 32A-B, 33A-D, and 34.

[0358] FIGS. 32A and 32B illustrate a flowchart of an example method 3200 for detecting a triggering event corresponding to placement of an item 204 on the platform 202, in accordance with one or more embodiments of the present disclosure. Method 3200 may be performed by item tracking device 104 as shown in FIG. 1. For example, one or more operations of method 3900 may be implemented, at least in part, in the form of software instructions (e.g., item tracking instructions 606 shown in FIG. 6), stored on tangible non-transitory computer-readable medium (e.g., memory 116 shown in FIGS. 1 and 6) that when run by one or more processors (e.g., processors 602 shown in FIG. 6) may cause the one or more processors to perform operations 3202-3230. It may be noted that method 3200 may be an alternative or additional embodiment to the process 700 described above with reference to FIG. 7 for detecting a triggering event. It may be noted that operations 3202-3230 are described primarily with reference to FIGS. 33A-33D, 34 and additionally with certain references to FIGS. 1, 2A, 16, and 17.

[0359] Referring to FIG. 32A, at operation 3202, item tracking device 104 captures a reference overhead depth image 3302 (shown in FIG. 33A), wherein the reference overhead depth image 3302 is captured by a 3D sensor 110 (shown in FIG. 2A) that is positioned above the platform 202 and that is configured to capture overhead depth images 124 (shown in FIGS. 33A-33D) of the platform 202. Each overhead depth image 124 taken by the 3D sensor 110 depicts a top view of the platform 202 including upward-facing surfaces of objects (e.g., items 204, a user's hand 3306 etc. shown in FIG. 33A-D.) placed on the platform 202. In one embodiment, the reference overhead depth image 3302 is of the platform 202 without any obstructions between the 3D sensor 110 and the surface 208 of the platform 202. In other words, the reference overhead depth image 3302 is of an empty platform 202 without any items 204 placed on the platform 202 and without any other objects (e.g., user's hand 3306 shown in FIGS. 33B and 33C) obstructing the view of the surface 208 of the platform 202 as viewed by the 3D sensor 110. In an additional or alternative embodiment, the reference overhead depth image 3302 is of the platform 202 with one or more items 204 already placed on the platform 202 as part of one or more previous interactions by the user with the imaging device 102.

[0360] As described above, the 3D sensor 110 is configured to capture depth images 124 such as depth maps or point cloud data for items 204 placed on the platform 202. A depth image 124 includes a plurality of pixels distributed across the depth image 124. Each of the plurality of pixels is associated with a depth value (d) 111 (shown in FIG. 2A) indicating a distance between the 3D sensor 110 and at least a portion of an upward-facing surface (e.g., surface 208 of the platform 202 or a surface of an object such as an item 204) depicted by the pixel.

[0361] At operation 3204, item tracking device 104 records (e.g., stores in memory 116 shown in FIG. 1) a reference depth value 3350 (also shown in FIG. 1) associated with the reference overhead depth image 3302. In one embodiment, the reference depth value 3350 includes a reference average depth value 3350a (shown in FIG. 1) associated with all pixels in the reference overhead depth image 3302. For example, item tracking device 104 calculates the reference depth value 3350 by adding individual depth values (d) 111 associated with all pixels in the reference overhead depth image 3302 and dividing the sum by the number of pixels in the reference overhead depth image 3302. In an alternative embodiment, the reference depth value 3350 includes the sum of depth values (d) 111 associated with all pixels in the reference overhead depth image 3302, wherein the item tracking device 104 calculates the reference depth value 3350 by adding depth values (d) 111 associated with all pixels of the reference overhead depth image 3302.

[0362] At operation 3206, item tracking device 104 monitors a proximity sensor 250 (shown in FIG. 2A) that is configured to detect motion near the platform 202 of the imaging device 102.

[0363] At operation 3208, if the item tracking device 104 does not detect motion near the platform 202 in conjunction with proximity sensor 250, the item tracking device 104 continues to monitor the proximity sensor 250 for motion. On the other hand, upon detecting motion near the platform 202 in conjunction with proximity sensor 250, method 3200 proceeds to operation 3210 where the item tracking device 104 starts capturing secondary overhead depth images 3304 (shown as 3304a in FIG. 33B, 3304b in FIG. 33C, and 3304c at FIG. 33D) of the platform 202. Motion detected near the platform 202 by the proximity sensor 250 may indicate that a user has approached the imaging device 102 and is about to place an item 204 on the platform 202.

[0364] From operation 3210, method 3200 proceeds in parallel to operations 3212 and 3216 (shown in FIG. 32B). At operation 3216, item tracking device 104 obtains the reference depth value 3350 that was previously stored in memory 116 at operation 3204.

[0365] At operation 3218, item tracking device 104 calculates a depth difference parameter (D) 3402 (shown in FIG. 34) based on a particular secondary overhead depth image 3304 (e.g., 3304a), for example, by comparing the secondary overhead depth image 3304 to the reference overhead depth image 3302. The depth difference parameter (D) 3402 is a single numerical value that represents a comparison of the depth values (d) 111 associated with pixels in the secondary overhead depth image 3304 and the pixels in the reference overhead depth image 3302. Item tracking device 104 calculates the depth difference parameter (D) 3402 by subtracting a depth value calculated based on individual depth values (d) 111 of pixels in the secondary overhead depth image 3304 from the reference depth value 3350 associated with the reference overhead depth image 3302.

[0366] In a first embodiment, when the reference depth value 3350 is a reference average depth value 3350a associated with pixels in the reference overhead depth image 3302, item tracking device 104 determines the depth difference parameter (D) 3402 by calculating a second average depth value associated with pixels in the secondary overhead depth image 3304 and subtracting the second average depth value from the reference average depth value 3350a. The second average depth value is an average of the individual depth values (d) 111 associated with pixels in the secondary overhead depth image 3304.

[0367] In a second alternative embodiment, when the reference depth value 3350 includes the sum of depth values (d) 111 associated with all pixels in the reference overhead depth image 3302, the item tracking device 104 determines the depth difference parameter (D) 3402 by subtracting a sum of depth values (d) 111 associated with pixels in the secondary overhead depth image 3304 from the sum of depth values (d) 111 associated with all pixels in the reference overhead depth image 3302.

[0368] While embodiments of the present disclosure are described with reference to the first embodiment described above wherein depth difference parameter (D) 3402 is determined by subtracting average of depth values (d) 111 associated with a secondary overhead depth image 3304 from the reference average depth value 3350a, a person having ordinary skill in the art may appreciate that these embodiments apply when the depth difference parameter (D) 3402 is determined by subtracting a sum of depth values (d) 111 associated with pixels in a secondary overhead depth image 3304 from the sum of depth values (d) 111 associated with all pixels in the reference overhead depth image 3302. Additionally, it may be noted that while certain embodiments of the present disclosure including FIGS. 33A, 33B, 33C, 33D and 34 illustrate and describe detecting a triggering event corresponding to placement of an item 204 on an empty platform, a person having ordinary skill in the art may appreciate that the embodiments apply to detecting a triggering event corresponding to placement of an additional item 204 on the platform 202 which has one or more other items 204 already placed on the platform 202.

[0369] Referring back to FIG. 32B, at operation 3220, if the depth difference parameter (D) 3402 is less than a predetermined threshold (Th) value (e.g., D<Th), method 3200 returns to operation 3218 where the item tracking device 104 continues to calculate the depth difference parameter (D) 3402 based on subsequent secondary overhead depth images 3304 captured by the 3D sensor 110. The value of the threshold (Th) may be set to a value slightly above zero to avoid false positives. It may be noted that D<Th indicates that there are no additional obstructions, as compared to the reference overhead depth image 3302, between the 3D sensor 110 and the surface 208 of the platform 202 at the time the secondary overhead depth image 3304 (based on which D was calculated) was captured. This means that the secondary overhead depth image 3304 is the same as or very similar to the reference overhead depth image 3302. For example, when the reference overhead depth image 3302 is of an empty platform 202 as shown in FIG. 33A, D<Th calculated by comparing a subsequently captured secondary overhead depth image 3304 and the reference overhead depth image 3302 indicates that the platform 202 was still empty at the time the subsequent secondary overhead depth image 3304 was taken, which may mean that the user has not initiated placement of an item 204 on the platform 202. For example, referring to FIG. 34, assuming that item tracking device 104 starts capturing secondary overhead depth images 3304 at t=0, the value of D stays below the pre-set threshold value (Th) between time t=0 and time t=t1. This indicates that the user's hand 3306 holding the item 204 does not enter the view of the 3D sensor 110 between time t=0 and time t=t1.

[0370] On the other hand, if the depth difference parameter (D) 3402 equals or is greater than the predetermined threshold (Th) value (e.g., D≥Th), method 3200 proceeds to operation 3222. D>Th indicates that there are one or more additional obstructions (e.g., user's hand 3306 and/or item 204), as compared to the reference overhead depth image 3302, between the 3D sensor 110 and the surface 208 of the platform 202 at the time the secondary overhead depth image 3304 (based on which D was calculated) was captured. For example, FIG. 33B shows a secondary overhead depth image 3304a which depicts a portion of the user's hand 3306 holding an item 204 (e.g., a soda can). The pixels in the secondary overhead depth image 3304a that depict the user's hand 3306 holding an item 204 are associated with a smaller average of depth value (d) 111 as compared to the reference average depth value 3350a corresponding pixels in the reference overhead depth image 3302. This means that the depth difference parameter (D) 3402 calculated as described above by comparing the secondary overhead depth image 3304a with the reference overhead depth image 3302 is a larger value as compared to a value of the depth difference parameter (D) 3402 calculated by comparing a previously captured secondary overhead depth image 3304 of an empty platform 202 with the reference overhead depth image 3302 which is also of the empty platform 202. A change in value of D from 0 or a value less than Th to a value greater than the Th, may indicate that the user has initiated the process of placing an item 204 on the platform 202 and that the user's hand 3306 holding an item 204 has moved from a first position that is outside a view of the 3D sensor 110 to a second position on the platform 202 (e.g., as shown in FIG. 33B) that is within the view of the 3D sensor 110. For example, as shown in FIG. 34, the value of D exceeds the pre-set threshold value (Th) at a secondary overhead depth image 3304 (e.g., 3304a) that is captured at time instant t1 (shown as event E1).

[0371] Referring back to FIG. 32B, at operation 3222, item tracking device 104 calculates a delta difference (ΔD) over a duration of a pre-set time interval (e.g., after time instant t1 in FIG. 34), wherein ΔD corresponds to a change in the depth difference parameter (D) 3402 over the pre-set time interval. For example, item tracking device 104 may calculate ΔD between time instant t2 and time instant t3 as follows:

[0372] $\Delta D=\frac{D2-D1}{t3-t2}$

• • [0373] where D1 is a depth difference parameter (D) 3402 calculated for a secondary overhead depth image 3304 captured at time instant t2; and
  • [0374] D2 is depth difference parameter (D) 3402 calculated for a secondary overhead depth image 3304 captured at time instant t3.

[0375] At operation 3224, if ΔD≥0, method 3200 returns to operation 3222 where item tracking device 104 continues to calculate ΔD based on subsequently captured secondary overhead depth images 3304. In one example, item tracking device 104 calculates ΔD each time the pre-set time interval has elapsed and based on secondary overhead depth images 3304 captured at either ends of the time interval. In a second example, item tracking device 104 may calculate ΔD periodically. In one embodiment, ΔD>0 indicates that the user's hand 3306 holding the item 204 is moving toward the platform 202 (e.g., inwards from an outer boundary of the platform 202). For example, as the user's hand 3306 holding the item 204 moves further inwards on the platform 202, depth values (d) 111 associated with more pixels of a secondary overhead depth image 3304 have smaller values (e.g., as compared depth values (d) 111 associated with pixels of a previously captured secondary overhead depth image 3304), causing the depth difference parameter D to be progressively larger. Thus, a positive change in the value of D (indicated by ΔD>0) over a pre-set time interval indicates that the user's hand 3306 holding the item 204 is moving toward the platform 202 (e.g., inwards from an outer boundary of the platform 202). For example, as shown in FIG. 34, the value of D increases between t1 and t2, causing ΔD>0, indicating that the user's hand 3306 holding the item 204 is moving toward the platform 202 (e.g., inwards from an outer boundary of the platform 202) between t1 and t2.

[0376] Referring back to FIG. 32B, if ΔD<0 (e.g., at operation 3224) over the pre-set time interval, method 3200 proceeds to operation 3226. For example, upon detecting that ΔD calculated based on two secondary overhead depth images 3304 on either ends of a pre-set time interval is less than zero, meaning that the value of the depth difference parameter (D) 3402 has dropped between the pre-set time interval, method 3200 proceeds to operation 3226. ΔD<0 may indicate that the user's hand 3306 has placed the item 204 on the platform 202 and is moving away from the platform 202. This is depicted in FIG. 33C. For example, as the user's hand 3306 moves away from the platform 202, depth values (d) 111 associated with more pixels of a secondary overhead depth image 3304 have larger values (e.g., as compared to pixels of a previously captured secondary overhead depth image 3304), causing the depth difference parameter D to be progressively smaller. Thus, a negative change in the value of D over a pre-set time interval indicates that the user's hand 3306 has placed the item 204 on the platform 202 and is moving away from the platform 202. For example, FIG. 33C shows a secondary overhead depth image 3304b which depicts the user's hand 3306 having placed the item 204 on the platform 202 and having moved away from the position of the item 204 on the platform 202. Referring to FIG. 34, item tracking device 104 calculates ΔD over the pre-set time interval between time instant t2 and time instant t3 based on secondary overhead depth images 3304 captured at t2 and t3. As shown, the value of the depth difference parameter D has dropped between t2 and t3, which means that ΔD<0 between t2 and t3.

[0377] Referring back to FIG. 32B, at operation 3226, item tracking device 104 calculates a delta difference ΔD over a subsequent pre-set time interval (e.g., after t3 in FIG. 34), wherein ΔD corresponds to a change in the depth difference parameter (D) 3402 over the subsequent pre-set time interval. It may be noted that the length of the subsequent pre-set time interval associated with operation 3226 may be different from the pre-set time interval associated with operation 3222 described above.

[0378] For example, item tracking device 104 may calculate ΔD between time t4 and time t5 as follows:

[0379] $\Delta D=\frac{D4-D3}{t5-t4}$

• • [0380] where D3 is a depth difference parameter calculated for a secondary overhead depth image 3304 captured at time instant t4; and
  • [0381] D4 is depth difference parameter calculated for a secondary overhead depth image 3304 captured at time instant t5.

[0382] At operation 3228, if ΔD≠0, method 3200 returns to operation 3226 where item tracking device 104 continues to calculate ΔD based on subsequently captured secondary overhead depth images 3304. In one example, item tracking device 104 calculates ΔD each time the subsequent pre-set time interval has elapsed and based on secondary overhead depth images 3304 captured at either ends of the subsequent pre-set time interval. In a second example, item tracking device 104 may calculate ΔD periodically. ΔD≠0 indicates that the user's hand 3306, after placing the item 204 on the platform 202, has not completely moved away from the platform 202 and out of the view of the 3D sensor 110. This means that one or more pixels of the secondary overhead depth images 3304 have captured at least a portion of the user's hand 3306, causing the depth difference parameter (D) 3402 to change between two secondary overhead depth images 3304 that were used to calculate ΔD.

[0383] Referring back to operation 3228 of FIG. 32B, if ΔD=0 (or near zero) over the subsequent pre-set time interval (e.g., between t4 and t5 in FIG. 34), method 3200 proceeds to operation 3230. For example, upon detecting that ΔD calculated based on two secondary overhead depth images 3304 on either ends of a subsequent pre-set time interval (e.g., between t4 and t5 in FIG. 34) equals zero (i.e., ΔD=0), meaning that the value of the depth difference parameter D was unchanged over the time interval, method 3200 proceeds to operation 3230. ΔD=0 may indicate that the user's hand 3306, after placing the item 204 on the platform 202, is out of view of the 3D sensor 110. This is depicted in FIG. 33D. Once the user's hand 3306 is out of view of the 3D sensor 110, depth values (d) 111 of pixels associated with a plurality of subsequent secondary overhead depth images 3304 remain unchanged, causing the depth difference parameter D to also stay unchanged. Thus, ΔD=0 detected after detecting ΔD<0 in a previous time interval may indicate that the user's hand 3306 has placed the item 204 on the platform 202 and has moved away far enough from the platform 202 that it is out of the view of the 3D sensor 110. For example, FIG. 33D shows a secondary overhead depth image 3304c which depicts the item 204 placed on the platform 202 with the user's hand 3306 out of view of the 3D sensor 110. Referring to FIG. 34, item tracking device 104 calculates ΔD over the pre-set time interval between time instant t4 and time instant t5 based on secondary overhead depth images 3304 captured at t4 and t5. As shown, the value of the depth difference parameter D is constant between t4 and t5, which means that ΔD=0 between t4 and t5.

[0384] Referring back to FIG. 32B, at operation 3230, in response to detecting that ΔD=0, item tracking device 104 determines that a trigger event has been detected, wherein the trigger event corresponds to placement of the item 204 on the platform. In other words, item tracking device 104 determines that the item 204 has been placed on the platform 202 and is ready for identification. In one embodiment, when item 204 is the first item to be placed on the platform 202 (e.g., on an empty platform 202), item tracking device 104 determines that a trigger event has been detected in response to detecting that ΔD=0 and that D>0. In other words. item tracking device 104 determines that a trigger event has been detected when D stays constant at a value higher than zero over a pre-set time interval.

[0385] In one or more embodiments, although the above discussion describes that the item tracking device 104 detects a triggering event in response to detecting events E1, E2 and E3 (shown in FIG. 34) one after the other in a sequence, it may be noted that the item tracking device 104 may detect a triggering event in response to detecting any one or more of these events. For example, item tracking device 104 may determine that a triggering event has been detected in response to detecting event E3 alone without detecting events E1 and E2.

[0386] In one or more embodiments, in response to determining that a trigger event has been detected, item tracking device 104 may initiate a procedure to identify the item 204 as described in certain embodiments of the present disclosure and display information associated with the identified item 204 on a user interface associated with the item tracking device 104.

[0387] Referring back to FIG. 32A, at operation 3212, in response to detecting a triggering event (e.g., at operation 3230 in FIG. 32B) or in response to detecting that a pre-set time interval has elapsed after initiating to capture the secondary overhead depth images 3304, item tracking device 104 stops capturing the secondary overhead depth images 3304. Operation then ends.

Identifying Unmoved Items on the Platform During Item Identification

[0388] In general, certain embodiments of the present disclosure describe techniques for detecting an item that was placed on the platform of the imaging device in a previous interaction and assigning to the item an item identifier that was identified in the previous interaction. The disclosed techniques determine whether an item has moved on the platform between interactions associated with a particular transaction. Upon determining that the item has not moved between interactions, the item is assigned an item identifier that was identified as part of a previous interaction. For example, when a first item is placed on the platform for the first time as part of an interaction, a first image of the first item is captured using an overhead camera positioned above the platform. An item identifier is determined for the first item and stored in a memory. Subsequently, when a second item is placed on the platform as part of a subsequent interaction, a second image of the first item is captured using the overhead camera. The second image of the first item is compared with the first image of the first item. When an overlap between the first and second images of the first item equals or exceeds a threshold, it is determined that the first item has not moved from its position on the platform between the first and second interactions. In response to determining that the first item has not moved between the two interactions, the first item is assigned the item identifier that was identified as part of the first interaction.

[0389] In certain embodiments, multiple items 204 may be placed on the platform 202 of the imaging device (shown in FIG. 2A) one-by-one for identification as part of a same transaction. For example, when purchasing a plurality of items 204 at a store where the imaging device 102 is deployed, a user may be instructed to place the items 204 on the platform 202 one by one for identification of the items 204. In this context, placement of each item 204 on the platform 202 as part of a particular transaction may be referred to as a separate interaction associated with the transaction. In response to detecting a triggering event corresponding to placement of an item 204 on the platform 202, item tracking device 104 may identify the item 204 using a method similar to the method described with reference to FIG. 23 and/or the method described with reference to FIG. 29. However, when a transaction includes multiple interactions, item tracking device 104 is configured to identify all items 204 placed on the platform 202 after each additional item 204 is placed on the platform 202. For example, when the user places a bottle of soda on the platform 202 as part of a first interaction, the item tracking device 104 identifies the bottle of soda. When the user adds a bag of chips on the platform 202 as part of a second interaction, the item tracking device 104 re-identifies the bottle of soda in addition to identifying the bag of chips that is newly placed on the platform 202. When the user adds a pack of gum to the platform 202 as part of a third interaction, the item tracking device again re-identifies the bottle of soda and re-identifies the bag of chips in addition to identifying the pack of gum that is newly placed on the platform 202. In other words, item tracking device 104 identifies all items 204 that are placed on the platform 202 as part of every interaction associated with a transaction despite the fact that all but one item 204 that was placed as part of a current interaction were already identified as part of previous interactions. This causes a lot of redundant processing as items 204 are re-identified as part of every subsequent interaction of the transaction. For example, FIG. 36A shows a first interaction 3620 of a transaction 3610, wherein the first interaction 3620 includes placement of a first item 204A on the platform 202. FIG. 36B shows a second interaction 3622 belonging to the same transaction 3610, wherein the second interaction 3622 includes placement of a second item 204B on the platform 202. As described above, item tracking device 104 is generally configured to identify the first item 204a as part of the first interaction 3620, and then re-identify the first item 204a along with identifying the second item 204B as part of the second interaction 3622.

[0390] Certain embodiments of the present disclosure describe improved techniques to identify items 204 placed on the platform 202 of an imaging device 102. As described below, these techniques retain item identifiers 1604 associated with items 204 that were identified as part of previous interactions to avoid re-identification of the same item 204. As described below, item tracking device 104 runs the item identification process (e.g., as described with reference to FIG. 23 and/or FIG. 29) only for that item 204 that was placed on the platform 202 as part of the latest interaction and that was not previously identified. For example, referring to the example discussed in the previous paragraph, when the user places a bottle of soda on the platform 202 as part of a first interaction, the item tracking device 104 identifies the bottle of soda and stores the identity of the bottle of soda in a memory. When the user adds a bag of chips on the platform 202 as part of a second interaction, the item tracking device 104 assigns the stored identity to the bottle of soda from the memory and only identifies the bag of chips that is newly placed on the platform 202 as part of the second interaction. The item tracking device 104 stores the identity of the bag of chips along with the identity of the bottle of soda as part of the second interaction. When the user adds a pack of gum to the platform 202 as part of a third interaction, the item tracking device 104 assigns the stored identities to the bottle of soda and the bag of chips from the memory, and only identifies the pack of gum that is newly placed on the platform 202 as part of the third interaction.

[0391] Thus, these techniques save processing resources associated with the item tracking device 104 that would otherwise be used in re-running item identification algorithms for items 204 that were already identified as part of a previous interaction of the transaction.

[0392] As described in more detail with reference to FIGS. 35A, 35B, 36A and 36B, item tracking device 104 determines whether an item 204 has moved on the platform 202 between interactions of a particular transaction. Upon determining that a particular item 204 has unmoved between interactions, item tracking device 104 assigns an item identifier to the particular item that was identified in a previous interaction. For example, referring to FIG. 36A in response to detecting that the first item 204A (e.g., a can of soda) has been placed on the platform 202 as part of the first interaction 3620, the item tracking device 104 captures a first overhead image of the platform 202 and identifies a first region 3612 within the first image 3632 that depicts the first item 204A. The item tracking device 104 identifies the first item 204A and stores a first item identifier 1604a associated with the first item 204A in a memory (e.g., memory 116 shown in FIG. 1). Referring to FIG. 36B, in response to detecting that a second item 204B (e.g., a bag of chips) has been added on the platform 202 as part of a second interaction 3622, item tracking device 104 captures a second overhead image 3634 of the platform 202 and determines a second region 3614 within the second image 3634 that depicts the first item 204A. The item tracking device 104 compares the second image 3634 to the first image 3632. In response to determining, based on the comparison, that an overlap between the first region 3612 of the first image 3632 and the second region 3614 of the second image 3634 equals or exceeds a threshold, item tracking device 104 accesses the first item identifier 1604a (e.g., from the memory 116) and assigns the first item identifier 1604a to the first item 204A depicted in the second image 3634. The overlap between the first region 3612 of the first image 3632 and the second region 3614 of the second image 3634 equaling or exceeding the threshold indicates that the first item 204A has unmoved on the platform 202 between the first interaction 3620 and the second interaction 3622. These aspects will now be described below in further detail with reference to FIGS. 35A-B, 36A-B, and 37.