Method, device, and storage medium for improving multi-object tracking

[0001] This application claims the priority benefit of Chinese Patent Application No. 202310405367.0, filed on Apr. 17, 2023 in the China National Intellectual Property Administration, the disclosure of which is incorporated herein in its entirety by reference.

[0002] The present disclosure relates generally to information processing and computer vision, and more particularly, to a method, a device and a storage medium for improving multi-object tracking.

[0003] With the development of computer science and artificial intelligence, it is becoming increasingly universal and effective to use computers to run artificial intelligence models based on neural networks to implement information processing. Computer vision is an important application field of artificial intelligence models.

[0004] A hotspot of computer vision technology is multi-object tracking. Multi-object tracking is commonly referred to as MTT (Multiple Target Tracking; sometimes also abbreviated as MOT: Multiple Object Tracking) briefly, which is used to detect and endow identifications (IDs) to objects of multiple types of interest such as pedestrians, automobiles or/or animals in a video, so as to perform trajectory tracking, without knowing the number of the objects in advance. A desired tracking result is that: for captured video segments, after multi-object tracking is performed, different objects are endowed with different IDs, so as to achieve work such as accurate tracking, accurate searching and the like. MOT is a key technology in the field of computer vision, and has been widely applied in fields such as autonomous driving, intelligent monitoring, behavior recognition and the like.

[0005] In multi-object tracking, for an input video, a tracking result of objects is output. In a tracking result image, each object is indicated by, for example, a rectangular bounding box with a corresponding ID identification number. In an image sequence of multiple frames of a video, a moving trajectory of a bounding box of the same ID can be regarded as a trajectory of an object of the ID. In these multiple frames, an image block sequence of multiple image blocks indicated by the bounding box of the ID is referred to as a tracklet (tracklet). In one tracklet, each image block therein can be regarded as a frame of image of the tracklet.

[0006] The adverse factors affecting the accuracy of a result of multi-object tracking include: occlusion, illumination, attitude changes, etc. It is challenging to improve the accuracy of multi-object tracking.

[0007] A brief summary of the present disclosure will be given below to provide a basic understanding of some aspects of the present disclosure. It should be understood that the summary is not an exhaustive summary of the present disclosure. It does not intend to define a key or important part of the present disclosure, nor does it intend to limit the scope of the present disclosure. The object of the summary is only to briefly present some concepts, which serves as a preamble of the detailed description that follows.

[0008] Existing multi-object tracking models may output imperfect tracklets. This is mainly reflected as two deficiencies: (1) one tracklet contains multiple different objects; (2) one object is in multiple different tracklets. The technical problems to be solved by the present disclosure include but are not limited to reducing at least one of the above-mentioned deficiencies, so as to improve the accuracy of multi-object tracking

[0009] According to an aspect of the present disclosure, there is provided a computer-implemented method for improving multi-object tracking. The method comprises: performing a split operation on each of a plurality of tracklets, for corresponding objects respectively, provided by a multi-object tracking model; wherein the split operation comprises: determining an appearance feature sequence corresponding to an image block sequence of the tracklet; determining a clustering label set by clustering a plurality of appearance features in the appearance feature sequence; determining, based on the clustering label set, an image block label sequence composed of clustering labels of appearance features of image blocks in the image block sequence; determining a fragment label sequence corresponding to continuous fragments, having the same clustering labels, in the image block label sequence, wherein each of fragment labels in the fragment label sequence is the same clustering label of a corresponding continuous fragment; in a case where a length of the fragment label sequence is greater than the number of types of the clustering labels in the clustering label set, updating the image block label sequence and the fragment label sequence by performing an update operation; and splitting the tracklet based on the updated image block label sequence; wherein the update operation comprises: updating the fragment label sequence based on a current continuous fragment corresponding to a current fragment label in the fragment label sequence, a previous continuous fragment corresponding to a previous fragment label previous to the current fragment label, and a next continuous fragment corresponding to a next fragment label following the current fragment label; and updating the image block label sequence based on the updated fragment label sequence.

[0010] According to an aspect of the present disclosure, there is provided a device for improving multi-object tracking. The device comprises: a memory, having instructions stored thereon; and at least one processor, coupled to the memory and configured to execute the instructions to perform a split operation on each of a plurality of tracklets, for corresponding objects respectively, provided by a multi-object tracking model; wherein the split operation comprises: determining an appearance feature sequence corresponding to an image block sequence of the tracklet; determining a clustering label set by clustering a plurality of appearance features in the appearance feature sequence; determining, based on the clustering label set, an image block label sequence composed of clustering labels of appearance features of image blocks in the image block sequence; determining a fragment label sequence corresponding to continuous fragments, having the same clustering labels, in the image block label sequence, wherein each of fragment labels in the fragment label sequence is the same clustering label of a corresponding continuous fragment; in a case where a length of the fragment label sequence is greater than the number of types of the clustering labels in the clustering label set, updating the image block label sequence and the fragment label sequence by performing an update operation; and splitting the tracklet based on the updated image block label sequence; wherein the update operation comprises: updating the fragment label sequence based on a current continuous fragment corresponding to a current fragment label in the fragment label sequence, a previous continuous fragment corresponding to a previous fragment label previous to the current fragment label, and a next continuous fragment corresponding to a next fragment label following the current fragment label; and updating the image block label sequence based on the updated fragment label sequence.

[0011] According to an aspect of the present disclosure, there is provided a computer-readable non-transitory storage medium storing a program thereon. The program, when executed by a computer, causes the computer to perform a split operation on each of a plurality of tracklets, for corresponding objects respectively, provided by a multi-object tracking model; wherein the split operation comprises: determining an appearance feature sequence corresponding to an image block sequence of the tracklet; determining a clustering label set by clustering a plurality of appearance features in the appearance feature sequence; determining, based on the clustering label set, an image block label sequence composed of clustering labels of appearance features of image blocks in the image block sequence; determining a fragment label sequence corresponding to continuous fragments, having the same clustering labels, in the image block label sequence, wherein each of fragment labels in the fragment label sequence is the same clustering label of a corresponding continuous fragment; in a case where a length of the fragment label sequence is greater than the number of types of the clustering labels in the clustering label set, updating the image block label sequence and the fragment label sequence by performing an update operation; and splitting the tracklet based on the updated image block label sequence; wherein the update operation comprises: updating the fragment label sequence based on a current continuous fragment corresponding to a current fragment label in the fragment label sequence, a previous continuous fragment corresponding to a previous fragment label previous to the current fragment label, and a next continuous fragment corresponding to a next fragment label following the current fragment label; and updating the image block label sequence based on the updated fragment label sequence.

[0012] The beneficial effects of the methods, devices and storage media of the present disclosure include: improving the accuracy of multi-object tracking.

[0021] Hereinafter, exemplary embodiments of the present disclosure will be described combined with the accompanying drawings. For the sake of clarity and conciseness, the specification does not describe all features of actual embodiments. However, it should be understood that many decisions specific to the embodiments may be made in developing any such actual embodiment, so as to achieve specific objects of a developer, and these decisions may vary as embodiments are different.

[0022] It should also be noted herein that, to avoid the present disclosure from being obscured due to unnecessary details, only those device structures closely related to the solution according to the present disclosure are shown in the accompanying drawings, while other details not closely related to the present disclosure are omitted.

[0023] It should be understood that, the present disclosure will not be limited only to the described embodiments due to the following description with reference to the accompanying drawings. Herein, where feasible, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment.

[0024] Computer program code for performing operations of various aspects of embodiments of the present disclosure can be written in any combination of one or more programming languages, the programming languages including object-oriented programming languages, such as Java, Smalltalk, C++ and the like, and further including conventional procedural programming languages, such as “C” programming language or similar programming languages.

[0025] Methods of the present disclosure can be implemented by circuitry having corresponding functional configurations. The circuitry includes circuitry for a processor.

[0026] An aspect of the present disclosure relates to a method for improving Multi-object Tracking (MOT). The method can be implemented with a computer. FIG. 1 illustrates an exemplary flowchart of a method 100 for improving multi-object tracking according to an embodiment of the present disclosure.

[0027] The method 100 comprises operation Op11 of: performing a split operation Op_sp on each (e.g., tracklet Trk[o]) of a plurality of tracklets (e.g., a tracklet Trk[oStart] to a tracklet Trk[oEnd]), for corresponding objects (e.g., an object Obj[oStart] to an object Obj[oEnd]) respectively, provided by a multi-object tracking model MOT-M. The method 100 can be regarded as a post-processing method of multi-object tracking. For a case where the tracklet Trk[oStart] to the tracklet Trk[oEnd] have been provided by the multi-object tracking model MOT-M, “oEnd-oStart+1” split operations can be performed. The multi-object tracking model MOT-M can determine tracklets of respective objects in a video based on input video segments. Any tracklet Trk[o] is represented by an image block sequence Seq_B[o]. The mage block sequence Seq_B[o] can be represented as: B[bStart], B[bStart+1], . . . , B[bEnd−1], B[bEnd]. Any image block B[b] in Seq_B[o] indicates an image of an object Obj[o] at a corresponding time. The mage block sequence Seq_B[o] is sorted in a time order. A time attribute value of the image block B[b] is a time of a corresponding video frame captured by a camera. The tracklet Trk[o] can come from either a single-camera based video segment or a multi-camera based video segment (that is, the MOT-M can be a multi-camera multi-object tracking model). It should be noted that: due to factors such as occlusion, illumination, attitude changes and the like, the tracklet Trk[o] provided by the model MOT-M may involve a case where it includes tracking sub-fragments of different objects that are not the object Obj[o], that is, multiple tracklets that were originally for multiple objects are determined as tracklets for one object. This reduces the accuracy of multi-object tracking, and thus in order to improve the performance of multi-object tracking, it is possible to perform a split operation Op_sp to attempt to split a single tracklet into multiple tracklets corresponding to the actual object number, so as to correct the aforementioned deficiencies.

[0028] As shown in FIG. 1, the split operation Op_sp may comprise: Operations Op101 to Op111.

[0029] In operation Op101, an appearance feature sequence Seq_Fa[o] corresponding to an image block sequence Seq_B[o] of the tracklet Tkr[o] is determined:

[0030] $\mathrm{Fa}\left[\mathrm{bStart}\right],\mathrm{Fa}\left[\mathrm{bStart}+1\right],\dots ,\mathrm{Fa}\left[\mathrm{bEnd}-1\right],\mathrm{Fa}\left[\mathrm{bEnd}\right].$

[0031] Appearance feature extraction is a routine operation of object tracking, which is implemented, for example, by utilizing a feature extraction part of a neural network. An appearance feature Fa[b] corresponds to an image block B[b]. The appearance feature can be provided by the MOT-M or by an independent feature extractor (e.g., a ReID model).

[0032] In operation Op103, a clustering label set cLS is determined by clustering a plurality of appearance features in the appearance feature sequence Seq_Fa[o]: {cL[cStart], . . . , cL[cEnd]}. The number N_cL of clustering classifications obtained through clustering can be represented as: N_cL=cEnd-cStart+1. A label of a clustering classification cL[c] can be represented by its index “c”. For example, for the convenience of discussion, when the clustering label set cLS has 5 classifications, the 5 classifications can be simply represented as C1, C2, C3, C4, C5.

[0033] In operation Op105, an image block label sequence Seq_bL composed of clustering labels of appearance features of image blocks in the image block sequence Seq_B[o] is determined based on the clustering label set cLS:

[0034] $B\left[\mathrm{bStart}\right].\mathrm{bL},B\left[\mathrm{bStart}+1\right].\mathrm{bL},\dots ,B\left[\mathrm{bEnd}-1\right].\mathrm{bL},B\left[\mathrm{bEnd}\right].\mathrm{bL}.$

[0035] Where, B[b].bL is a label of a clustering classification for the appearance feature Fa[b] of the image block B[b], which is abbreviated as an “image block label”. For example, if the clustering label Fa[b].cL of Fa[b] is “C1”, then the image block label B[b].bL of the image block B[b] is “C1”.

[0036] In operation Op107, a fragment label sequence Seq_fL (e.g., fL[fStart], . . . , fL[fEnd]) corresponding to continuous fragments, having the same clustering labels, in the image block label sequence Seq_bL, is determined, wherein each fL[f] of fragment labels in the fragment label sequence Seq_fL is the same clustering label of a corresponding continuous fragment Fg[f]. For example, if an image block label of each image block in continuous fragments B[1] to B[32] in Seq_bL is “C1” and image block labels of B[33] to B[128] are all “C2”, then the image blocks B[1] to B[32] form a continuous fragment Fg[1], the image block fragments B[33].bL to B[128].bL form a continuous fragment Fg[2], a label (i.e., fragment label fL[1]) of the fragment Fg[1] is C1, a label (i.e., fragment label fL[2]) of the fragment Fg[2] is C2, and the fragment label sequence Seq_FL is: C1, C2. This has been shown in Table 1.

[0037]

TABLE 1
Seq_B	B[1]	. . .	B[32]	B[33]	. . .	B[128]	. . .
Seq_Fa	Fa[1]	. . .	Fa[32]	Fa[33]	. . .	Fa[128]	. . .
Fa.cL	C1	. . . (C1)	C1	C2	. . . (C2)	C2	C3, . . .
B.bL	C1	. . . (C1)	C1	C2	. . . (C2)	C2	C3, . . .
(Seq_bL)
Fg	Fg[1]: B[1].bL to B[32].bL	Fg[2]: B[33].bL to B128].bL	Fg[3], . . .
	(a sequence of 32 labels	(a sequence of 96 labels “C2”)
	“C1”)
fL	C1	C2	C3, . . .
(Seq_fL)

[0038] Each continuous fragment Fg is a fragment (i.e., a subsequence) of the sequence Seq_bL, and a “start position” attribute (Fg.start) and an “end position” attribute (Fg.end) can be provided for each Fg to record start and end positions of Fg, so that each member in Fg is accessible in a case where an index of Fg is known. If a certain Fg is updated, it means that Seq_BL is also updated.

[0039] In operation Op109, in a case where a length N_fL of the fragment label sequence Seq_fL is greater than the number N_cL of types of the clustering labels in the clustering label set cLS, the image block label sequence Seq_bL and the fragment label sequence Seq_fL are updated by performing an update operation Op_ud. The length N_fL is the number of the continuous fragments Fg of the sequence Seq_bL, i.e., the member number of the sequence Seq_fL.

[0040] The update operation Op_ud comprises: operations Op1091 and Op1093. In operation Op1091, the fragment label sequence Seq_fL is updated based on a current continuous fragment cFg corresponding to a current fragment label cfL in the fragment label sequence, a previous continuous fragment pFg corresponding to a previous fragment label pfL previous to the current fragment label, and a next continuous fragment nFg corresponding to a next fragment label nfL following the current fragment label (the “update” here means changing fragment labels at relevant positions under reasonable conditions). In operation Op1093, the image block label sequence Seq_bL is updated based on the updated fragment label sequence (the “update” here means changing fragment labels at relevant positions under reasonable conditions).

[0041] In operation Op111, the tracklet Trk[o] is split based on the updated image block label sequence. For example, if the updated image block label sequence includes 128 image block labels, first to 64-th image block labels (clustering labels) are “C11” and 65-th to 128-th image block labels are “C12”, then at a position between the 64-th image block and the 65-th image block, the image block sequence of the tracklet Trk[o] is split into two image block sequences corresponding to different objects, that is, the tracklet Trk[o] is split into two tracklets.

[0042] The image block sequence Seq_B[o] may contain some low-quality image blocks, which are disadvantageous to obtaining a better split result. In this regard, it is possible to consider performing appropriate filtering on image blocks. In an embodiment, the split operation Op_sp in the method 100 further comprises: updating the clustering label set by removing a noise cluster Cns in the clustering label set cLS; and updating the image block sequence by removing image blocks in the image block sequence Seq_B[o] which correspond to the noise cluster. The noise cluster Cns satisfies: the number of the corresponding image blocks of the noise cluster is lower than a predetermined threshold, and an average image block detection score of the corresponding image blocks is lower than a first score threshold. The number of noise clusters in the clustering label set cLS may be zero, 1 or more. An image block corresponding to a noise cluster may be referred to as a noise image block, which is disadvantageous to determining accurate split points, and thus the noise image block is removed. After the noise image blocks are removed, operation Op105 and subsequent operations are then performed.

[0043] In an embodiment, the update operation Op_ud of the method 100 comprises: in a case where the previous fragment label is same as the next fragment label (i.e., pfL=nfL), if a length cFg.Le of the current continuous fragment is less than or equal to a first length threshold (Length1), updating the current fragment label to the previous fragment label, and updating all clustering labels in the current continuous fragment to the previous fragment label; and otherwise, updating the previous fragment label to a new clustering label (i.e., a clustering label absent in the clustering label set), updating all clustering labels in the previous continuous fragment to the new clustering label, and updating the clustering label set by adding the new clustering label into the clustering label set. An example of changes in the labels before and after the update is as shown in Table 2.

[0044]

	TABLE 2
	fL[i − 1] (pfL)	fL[i] (cfL)	fL[i + 1] (nfL)
Before update	C1	C2	C1
(N_cL = 12)
cFg.Len ≤ Lenth1	C1	C1	C1
after the update
cFg.Len > Lenth1	C13	C2	C1
after the update
(N_cL = 13)

[0045] In an embodiment, the update operation Op_ud of the method 100 comprises: in a case where the previous fragment label pfL is different from the next fragment label nfL, keeping image block labels of the previous continuous fragment, the current continuous fragment, and the next continuous fragment unchanged.

[0046] In an embodiment, the update operation Op_ud of the method 100 comprises: in a case of selecting each of fragment labels from a second fragment label to a second to last fragment label in the fragment label sequence as the current fragment label respectively, determining whether to update the fragment label sequence. After the current fragment label traverses each of the fragment labels, from the second fragment label to the second to last fragment label, in the fragment label sequence, in a case where it is determined that no fragment label update operation occurs during the traversing operation, a fragment label, in the fragment label sequence, that appears for a second time relative to a previous fragment label sequence is replaced with a new fragment label. At the same time, the new fragment label is added into the clustering label set so as to update the clustering label set.

[0047] FIG. 2 illustrates an exemplary flowchart of a method 200 for an update operation Op_ud according to an embodiment of the present disclosure.

[0048] In operation Op201, it is determined whether N_fL is greater than N_cL. If a determination result is “yes” (represented by “Y” in the figure), then operation Op202 is performed; if a determination result is “no” (represented by “N” in the figure), then the method 200 ends.

[0049] In operation Op202, an index i that records the current fragment label is set to 2, and an update flag Ch is set to “0” (“O” indicates that no update occurs).

[0050] In operation Op203, it is determined whether the current fragment label fL[i−1] is same as the next fragment label fL[i+1]. If a determination result is “Y”, then operation Op204 is performed; if a determination result is “N”, then operation Op207 is performed.

[0051] In operation Op204, it is determined whether a length Fg[i]. Len of the current continuous fragment is less than or equal to a predetermined threshold Length1. If a determination result is “Y”, then operation Op206 is performed; if a determination result is “N”, then operation Op205 is performed.

[0052] In operation Op205, a new clustering label is set for fL[i−1], the clustering label of fL[i−1] is updated, cLS is updated, and Ch is endowed with a value 1 (“1” indicates that an update occurs). The set new clustering label is a clustering label absent in the clustering label set cLS. For example, if labels in cLS currently are numbered to “C30”, then the new clustering label may be C31″. Since Fg[i−1] is a subsequence of Seq_bL, updating the clustering label of Fg[i−1] means that Seq_bL is updated.

[0053] In operation Op206, fL[i] is updated to fL[i−1], the clustering label of Fg[i] is updated, and Ch is endowed with a value 1 (“1” indicates that an update occurs). Updating the clustering label of Fg[i] means that Seq_bL is updated.

[0054] In operation Op207, the index i of the current fragment label is added by 1.

[0055] In operation Op208, it is determined whether the index i of the current fragment label is equal to N-fL. If a determination result is “Y”, then operation Op209 is performed; if a determination result is “N”, then operation Op203 is performed.

[0056] In operation Op209, it is determined whether Ch is “0”. That is, it is determined whether the fragment label is changed in a process during which i traverses the fragment labels from the second fragment label to the second to last fragment label. If a determination result is “Y”, then operation Op210 is performed; if a determination result is “N”, then operation Op211 is performed.

[0057] In operation Op210, a fragment label, in the fragment label sequence, that appears for a second time relative to a previous fragment label sequence is replaced with a new fragment label. For example, for Seq_fL: C4, C2, C3, C4, C1, C5, C4, after replacing, an updated Seq_fLis: C4, C2, C3, C6, C1 C5, C7.

[0058] In operation Op211, Seq_Fg, Seq_bL, cLS, N_fL, N_cL are updated based on the current Seq_fL. After operation Op211 is performed, the process returns to operation Op201. cLS is updated so that cLS is composed of fragment labels in Seq_fL. N_fL is updated so that N_fL is equal to the number of members in the current Seq_fL. N-cL is updated so that N_cL is equal to the number of elements in the updated cLS.

[0059] To solve the problem that one object exists in multiple different tracklets, the method for improving multi-object tracking of the present disclosure thus can further comprise: performing a merge operation on a plurality of updated tracklets obtained by performing the splitting operation on the plurality of tracklets.

[0060] FIG. 3 illustrates an exemplary flowchart of a method 300 for improving multi-object tracking according to an embodiment of the present disclosure. The method 300 comprises operations Op11 and Op33. In other words, the method 100 can also comprise operation Op33.

[0061] Operation Op11 is same as the operation in the method 100: performing a split operation Op_sp on each (e.g., tracklet Trk[o]) of a plurality of tracklets (e.g. tracklet Trk[oStart] to tracklet Trk[oEnd]), for corresponding objects (e.g., object Obj[oStart] to object Obj[oEnd]) respectively, provided by a multi-object tracking model MOT-M.

[0062] In operation Op33, a merge operation Op_mg is performed on a plurality of updated tracklets obtained by performing the splitting operation on the plurality of tracklets. As shown in FIG. 3, the merge operation Op_mg comprises operations Op331 to Op336.

[0063] In operation Op331, a first cosine distance matrix W of the plurality of updated tracklets is determined based on a first type of appearance features of respective updated tracklets associated with image block detection scores.

[0064] In operation Op332, a second cosine distance matrix D of the plurality of updated tracklets is determined based on a second type of appearance features of respective updated tracklets associated with most similar image block pairs between tracklets.

[0065] In operation Op333, a Hadamard product of the first cosine distance matrix W and the second cosine distance matrix D is determined as a third cosine distance matrix H.

[0066] In operation Op334, based on information of an image block pair for an element in the third cosine distance matrix H, the element is adjusted.

[0067] In operation Op335, hierarchical clustering is performed on the plurality of updated tracklets based on the adjusted third cosine distance matrix

[0068] In operation Op336, a plurality of updated tracklets belonging to the same hierarchical clustering classification are merged based on a clustering result of the hierarchical clustering.

[0069] In an example, a first type of appearance feature of a current updated tracklet is a weighted average value of a result obtained by performing weighted averaging on appearance features of corresponding image blocks using image block detection scores; and in performing the weighted summation, the weighted summation is only performed on appearance features of image blocks having image block detection scores greater than or equal to a second score threshold. For example, a first type of appearance feature f_Ti of an updated tracklet T_i is determined according to equation (1).

[0070] $\begin{array}{cc}{f}_{T_i}=\frac{1}{N_i}{\sum }_{k=1}^{N_i}\left(s_k^i{f}_k^i\right),s_k^i\ge s_{\mathrm{th}}& \left(1\right)\end{array}$

[0071] Where, S_th is a predetermined second score threshold,

[0072] $S_k^i$
is a detection score of a k-th image block in the tracklet T_i,

[0073] $f_k^i$
is an appearance feature of the k-th image block, and N_i is the number of image blocks satisfying detection score conditions. S_th can take a larger value, so as to select image blocks of higher quality (high credibility, high importance) to determine a first type of appearance feature of a tracklet. The appearance feature and the detection score can be either given by the model MOT-M or respectively provided by an independent feature extractor or detector. Correspondingly, the element W_i,j of the first cosine distance matrix W can be represented as equation (2).

[0074] $\begin{array}{cc}{w}_{i,j}=1-\cos \left(f_{T_i},f_{T_j}\right)& \left(2\right)\end{array}$

[0075] Where,

[0076] $\cos \left(f_{T_i},f_{T_j}\right)$
is a cosine similarity between the appearance features

[0077] $f_{T_i}\mathrm{and}{f}_{T_j}.$

[0078] In an example, referring to equation (3), an element d_i,j, in the second cosine distance matrix D and associated with a first tracklet T_i and a second tracklet T_j in the plurality of updated tracklets, is an average value of cosine distances of appearance feature pairs of a plurality of most similar image block pairs between the first tracklet and the second tracklet.

[0079] $\begin{array}{cc}{d}_{i,j}=\frac{1}{M}{\sum }_{m=1}\left(1-\cos \left(f_m^i,f_m^j\right)\right)& \left(3\right)\end{array}$

[0080] Where, it is possible to calculate a cosine similarity between an image block in the tracklet T_i and an image block in the tracklet T_j (the two image blocks form an image pair) based on appearance features of the image blocks, and after the obtained cosine similarities are arranged in a descending order, appearance feature pairs of a plurality of M most similar image block pairs corresponding to top M cosine similarities are taken to calculate the element d_i,j.

[0081] Based on information of an image block pair for an element h_i,j in the third cosine distance matrix H, adjusting the element comprises:

• • [0082] (1) if the number of overlapping image blocks of two corresponding tracklets of the element is greater than a first overlapping threshold, adjusting the element to a first value greater than or equal to 1;
  • [0083] (2) if an absolute value of a time interval between two corresponding tracklets of the element is greater than a first time threshold, adjusting the element to a second value greater than or equal to 1;
  • [0084] (3) if the element h_i,j is a diagonal element of the third cosine distance matrix H, adjusting the element to a third value greater than or equal to 1.

[0085] The first, second and third values are, for example, 1, a value close to 2 or greater. The first, second and third values can be equal.

[0086] In an example, based on a clustering result of the hierarchical clustering, merging a plurality of updated tracklets belonging to the same hierarchical clustering classification comprises: connecting a plurality of updated tracklets belonging to the same hierarchical clustering classification, in a time order, as a merged tracklet; determining whether a case where there are a plurality of image blocks for the same time exists in an image block sequence of the merged tracklet; and in a case where a determination result is “yes”, performing the following operation to update the merged tracklet; removing, from the image block sequence, other image blocks than an image block with a highest detection score among the plurality of image blocks.

[0087] The update operation will be described by way of examples below.

[0088] FIG. 4 illustrates a schematic diagram of changes in an image block label sequence according to an embodiment of the present disclosure. As shown in FIG. 4, an initial image block label sequence Seq_bL contains two types of image block labels distinguished by different colors: C1 and C2; according to changes in the labels, Seq_bL is divided into 6 sub-fragments, namely 6 continuous fragments Fg[1], Fg[2], Fg[3], Fg[4], Fg[5], Fg[6]; an illustrated length of each continuous fragment is proportional to the number (i.e., the number of image blocks in a corresponding image block sub-fragment) of labels the continuous fragment contains. According to the method 200, in a process during which i traverses from 2 to 5: when i=2, since the length of the continuous fragment Fg[2] is less than the threshold Length1, a fragment label update occurs and all labels “C2” thereof are replaced with “C1”; when i=5, since the length of the continuous segment Fg[5] is less than the threshold Length1, a fragment label update occurs and all labels “C1” thereof are replaced with “C2”.

[0089] FIG. 5 illustrates another schematic diagram of changes in an image block label sequence according to an embodiment of the present disclosure. As shown in FIG. 5, an initial image block label sequence Seq_bL contains three types of image block labels distinguished by different colors: C1, C2, C3; according to changes in the labels, Seq_bL is divided into 6 sub-fragments, namely 6 continuous fragments Fg[1], Fg[2], Fg[3], Fg[4], Fg[5], Fg[6]; an illustrated length of each continuous fragment is proportional to the number (i.e., the number of image blocks in a corresponding image block sub-fragment) of labels the continuous fragment contains. According to the method 200, in a process during which i traverses from 2 to 5: when i=2, since the continuous fragment Fg[2] is long enough and labels of two fragments previous and next to Fg[2] are the same, a fragment label update occurs and all labels “C1” of the fragment Fg[1] are replaced with a new label “C4”; when i=5, since the length of the continuous segment Fg[5] is less than the threshold Length1, a fragment label update occurs and all labels “C1” thereof are replaced with “C2”.

[0090] Effects of the present disclosure will be described below.

[0091]

TABLE 3
	Initial	Tracklet number after	Tracklet number after
	tracklet	splitting by baseline	splitting by
Video	number	method	method 100
MOT17-01	49	127	58
MOT17-03	199	438	223
MOT17-06	202	572	263

[0092] The effects of the method 100 are tested using a dataset MOT17 of a public testing platform in MOT field, with testing results being as shown in Table 3, wherein the baseline method is a “ReMOT” method (Yang F, Chang X, Sakti S, et al., ReMOT: A model-agnostic refinement for multiple object tracking; Image and Vision Computing, 2021, 106:104091.). The initial tracklet number is obtained through calculation by a “ByteTrack” model. With respect to the baseline method, the split method of the present disclosure can better suppress an “over-split”. That is, it can effectively reduce false split points. Sources of these false split points include, but are not limited to, an object being occluded for a short time or a suddenly-turning object.

[0093]

TABLE 4
Dataset	Method	IDF1(%)	HOTA (%)	IDSW
MOT17	Benchmark method	77.3	63.1	2,196
	Method 300	82.2 (+4.9)	65.4 (+2.3)	1,497 (−699)
MOT20	Benchmark method	75.2	61.3	1,223
	Method 300	78.2 (+3.0)	63.0 (+1.7)	1,025 (−198)

[0094] The effects of the method 300 are tested using datasets MOT17 and MOT20 of a public testing platform in MOT field, with testing results being as shown in Table 4, wherein the baseline method is a “ByteTrack” model. With respect to the baseline method, the method of the present disclosure has been somewhat improved in terms of all three performance characterization parameters. The testing demonstrates that operations of the present disclosure can make tracklets have better one-to-one correspondence to objects.

[0095] In an embodiment of the present disclosure, there is provided a device for improving multi-object tracking. Exemplary description will be made with reference to FIG. 6 below. FIG. 6 illustrates an exemplary block diagram of a device 600 for improving multi-object tracking according to an embodiment of the present disclosure. The device 600 comprises: a memory 601, having instructions Inst stored thereon; and at least one processor 603, coupled to the memory 601 and configured to execute the instructions to perform a split operation on each of a plurality of tracklets, for corresponding objects respectively, provided by a multi-object tracking model; wherein the split operation comprises: determining an appearance feature sequence corresponding to an image block sequence of the tracklet; determining a clustering label set by clustering a plurality of appearance features in the appearance feature sequence; determining, based on the clustering label set, an image block label sequence composed of clustering labels of appearance features of image blocks in the image block sequence; determining a fragment label sequence corresponding to continuous fragments, having the same clustering labels, in the image block label sequence, wherein each of fragment labels in the fragment label sequence is the same clustering label of a corresponding continuous fragment; in a case where a length of the fragment label sequence is greater than the number of types of the clustering labels in the clustering label set, updating the image block label sequence and the fragment label sequence by performing an update operation; and splitting the tracklet based on the updated image block label sequence; wherein the update operation comprises: updating the fragment label sequence based on a current continuous fragment corresponding to a current fragment label in the fragment label sequence, a previous continuous fragment corresponding to a previous fragment label previous to the current fragment label, and a next continuous previous to corresponding to a next fragment label following the current fragment label; and updating the image block label sequence based on the updated fragment label sequence. The instructions Inst have a corresponding relationship with the method 100. For the further configuration of the device 600, reference may be made to the description of the method 100 of the present disclosure. The instructions Inst can further comprise instructions that correspond to the merge operation, that is, the instructions Inst can also correspond to the method 300.

[0096] An aspect of the present disclosure provides a computer-readable non-transitory storage medium storing a program thereon, wherein the program, when executed by a computer, causes the computer to perform a split operation on each of a plurality of tracklets, for corresponding objects respectively, provided by a multi-object tracking model; wherein the split operation comprises: determining an appearance feature sequence corresponding to an image block sequence of the tracklet; determining a clustering label set by clustering a plurality of appearance features in the appearance feature sequence; determining, based on the clustering label set, an image block label sequence composed of clustering labels of appearance features of image blocks in the image block sequence; determining a fragment label sequence corresponding to continuous fragments, having the same clustering labels, in the image block label sequence, wherein each of fragment labels in the fragment label sequence is the same clustering label of a corresponding continuous fragment; in a case where a length of the fragment label sequence is greater than the number of types of the clustering labels in the clustering label set, updating the image block label sequence and the fragment label sequence by performing an update operation; and splitting the tracklet based on the updated image block label sequence; wherein the update operation comprises: updating the fragment label sequence based on a current continuous fragment corresponding to a current fragment label in the fragment label sequence, a previous continuous fragment corresponding to a previous fragment label previous to the current fragment label, and a next continuous fragment corresponding to a next fragment label following the current fragment label; and updating the image block label sequence based on the updated fragment label sequence. The program has a corresponding relationship with the method 100. For the further configuration of the program, reference may be made to the description of the method 100 of the present disclosure. The program can further comprise a portion that corresponds to the merge operation, that is, the program can also correspond to the method 300.

[0097] According to an aspect of the present disclosure, there is further provided an information processing apparatus.

[0098] The present disclosure also relates to an information processing apparatus. FIG. 7 is an exemplary block diagram of an information processing apparatus 700 according to an embodiment of the present disclosure. In FIG. 7, a Central Processing Unit (CPU) 701 executes various processing according to programs stored in a Read-Only Memory (ROM) 702 or programs loaded from a storage device 708 to a Random Access Memory (RAM) 703. In the RAM 703, data needed when the CPU 701 executes various processing and the like is also stored as needed.

[0099] The CPU 701, the ROM 702 and the RAM 703 are connected to each other via a bus 704. An input/output interface 705 is also connected to the bus 704.

[0100] The following components are connected to the input/output interface 705: an input device 706, including a soft keyboard and the like; an output device 707, including a display such as a Liquid Crystal Display (LCD) and the like, as well as a speaker and the like; the storage device 708 such as a hard disc and the like; and a communication device 709, including a network interface card such as an LAN card, a modem and the like. The communication device 709 executes communication processing via a network such as the Internet, a local area network, a mobile network or a combination thereof.

[0101] A driver 710 is also connected to the input/output interface 705 as needed. A removable medium 711 such as a semiconductor memory and the like is installed on the driver 710 as needed, such that programs read therefrom are installed in the storage device 708 as needed.

[0102] The CPU 701 can run a program corresponding to a method for improving multi-object tracking.

[0103] The beneficial effects of the methods, devices and storage media of the present disclosure include at least of: improving the accuracy of multi-object tracking. Specifically, the methods, devices and storage media of the present disclosure can be used to perform post-processing on a tracklet set provided by an MOT-M and output an updated tracklet set which has improved accuracy. The split operation of the present disclosure can reduce a case where one tracklet includes multiple different objects and, as compared with other split methods, can effectively reduce a false split and suppress an over-split. The merge operation of the present disclosure can solve a problem that one object exists in multiple different tracklets. For an embodiment where a split operation and a merge operation are simultaneously included, an improvement effect will be more apparent, and the accuracy of an updated tracklet set is obviously improved. For the merge operation of the present disclosure, it not only simply considers average appearance features but also further considers detection credibility (i.e., detection scores of detection boxes) of different image blocks in one segment of trajectory, and it also considers distances between most similar detection block pairs (i.e., the most similar image block pairs) in two segments of trajectory. Therefore, it is advantageous to improving a merge effect.

[0104] As described above, according to the present disclosure, the principle of improving multi-object tracking has been disclosed. It should be noted that, the effects of the solution of the present disclosure are not necessarily limited to the above-mentioned effects, and in addition to or instead of the effects described in the preceding paragraphs, any of the effects as shown in the specification or other effects that can be understood from the specification can be obtained.

[0105] Although the present invention has been disclosed above through the description with regard to specific embodiments of the present invention, it should be understood that those skilled in the art can design various modifications (including, where feasible, combinations or substitutions of features between various embodiments), improvements, or equivalents to the present invention within the spirit and scope of the appended claims. These modifications, improvements or equivalents should also be considered to be included within the protection scope of the present invention.

[0106] It should be emphasized that, the term “comprise/include” as used herein refers to the presence of features, elements, steps or assemblies, but does not exclude the presence or addition of one or more other features, elements, steps or assemblies.

[0107] In addition, the methods of the various embodiments of the present invention are not limited to be executed in the time order as described in the specification or as shown in the accompanying drawings, and may also be executed in other time orders, in parallel or independently. Therefore, the execution order of the methods as described in the specification fails to constitute a limitation to the technical scope of the present invention.

[0108] The present disclosure includes but is not limited to the following solutions.

[0109] 1. A computer-implemented method for improving multi-object tracking, characterized by comprising: performing a split operation on each of a plurality of tracklets, for corresponding objects respectively, provided by a multi-object tracking model;

• • [0110] wherein the split operation comprises:
    • [0111] determining an appearance feature sequence corresponding to an image block sequence of the tracklet;
    • [0112] determining a clustering label set by clustering a plurality of appearance features in the appearance feature sequence;
    • [0113] determining, based on the clustering label set, an image block label sequence composed of clustering labels of appearance features of image blocks in the image block sequence;
    • [0114] determining a fragment label sequence corresponding to continuous fragments, having the same clustering labels, in the image block label sequence, wherein each of fragment labels in the fragment label sequence is the same clustering label of a corresponding continuous fragment;
    • [0115] in a case where a length of the fragment label sequence is greater than the number of types of the clustering labels in the clustering label set, updating the image block label sequence and the fragment label sequence by performing an update operation; and
    • [0116] splitting the tracklet based on the updated image block label sequence;
  • [0117] wherein the update operation comprises:
    • [0118] updating the fragment label sequence based on a current continuous fragment corresponding to a current fragment label in the fragment label sequence, a previous continuous fragment corresponding to a previous fragment label previous to the current fragment label, and a next continuous fragment corresponding to a next fragment label following the current fragment label; and
    • [0119] updating the image block label sequence based on the updated fragment label sequence.

[0120] 2. The method according to Appendix 1, wherein the update operation comprises:

• • [0121] in a case where the previous fragment label is same as the next fragment label,
    • [0122] if a length of the current continuous fragment is less than or equal to a first length threshold, updating the current fragment label to the previous fragment label, and updating all clustering labels in the current continuous fragment to the previous fragment label; and
    • [0123] otherwise, updating the previous fragment label to a new clustering label, updating all clustering labels in the previous continuous fragment to the new clustering label, and updating the clustering label set by adding the new clustering label into the clustering label set.

[0124] 3. The method according to Appendix 2, wherein the update operation comprises:

• • [0125] in a case where the previous fragment label is different from the next fragment label, keeping image block labels of the previous continuous fragment, the current continuous fragment, and the next continuous fragment unchanged.

[0126] 4. The method according to Appendix 2, wherein the update operation comprises:

• • [0127] in a case of selecting each of fragment labels from a second fragment label to a second to last fragment label in the fragment label sequence as the current fragment label respectively, determining whether to update the fragment label sequence.

[0128] 5. The method according to Appendix 4, wherein the update operation comprises:

• • [0129] after the current fragment label traverses each of the fragment labels from the second fragment label to the second to last fragment label in the fragment label sequence, in a case where it is determined that no fragment label update operation occurs during the traversing operation,
    • [0130] replacing a fragment label, in the fragment label sequence, that appears for a second time relative to a previous fragment label sequence with a new fragment label.

[0131] 6. The method according to Appendix 1, wherein the split operation comprises:

• • [0132] updating the clustering label set by removing a noise cluster in the clustering label set; and
  • [0133] updating the image block sequence by removing image blocks in the image block sequence which correspond to the noise cluster; and
  • [0134] the noise cluster satisfies: the number of the corresponding image blocks of the noise cluster is lower than a predetermined threshold, and an average image block detection score of the corresponding image blocks is lower than a first score threshold.

[0135] 7. The method according to Appendix 1, characterized by further comprising: performing a merge operation on a plurality of updated tracklets obtained by performing the splitting operation on the plurality of tracklets;

• • [0136] wherein the merge operation comprises:
    • [0137] determining a first cosine distance matrix of the plurality of updated tracklets based on a first type of appearance features of respective updated tracklets associated with image block detection scores;
    • [0138] determining a second cosine distance matrix of the plurality of updated tracklets based on a second type of appearance features of respective updated tracklets associated with most similar image block pairs between tracklets;
    • [0139] determining a Hadamard product of the first cosine distance matrix and the second cosine distance matrix as a third cosine distance matrix;
    • [0140] based on information of an image block pair for an element in the third cosine distance matrix, adjusting the element;
    • [0141] performing hierarchical clustering on the plurality of updated tracklets based on the adjusted third cosine distance matrix; and
    • [0142] based on a clustering result of the hierarchical clustering, merging a plurality of updated tracklets belonging to the same hierarchical clustering classification.

[0143] 8. The method according to Appendix 7, wherein a first type of appearance feature of a current updated tracklet is a weighted average value of a result obtained by performing weighted averaging on appearance features of corresponding image blocks using image block detection scores; and

• • [0144] in performing the weighted summation, the weighted summation is only performed on appearance features of image blocks having image block detection scores greater than or equal to a second score threshold.

[0145] 9. The method according to Appendix 7, wherein an element, in the second cosine distance matrix, associated with a first tracklet and a second tracklet in the plurality of updated tracklets, is an average value of cosine distances of appearance feature pairs of a plurality of most similar image block pairs between the first tracklet and the second tracklet.

[0146] 10. The method according to Appendix 7, wherein based on information of an image block pair for an element in the third cosine distance matrix, adjusting the element comprises:

• • [0147] if the number of overlapping image blocks of two corresponding tracklets for the element is greater than a first overlapping threshold, adjusting the element to a first value greater than or equal to 1.

[0148] 11. The method according to Appendix 7, wherein based on information of an image block pair for an element in the third cosine distance matrix, adjusting the element comprises:

• • [0149] if an absolute value of a time interval between two corresponding tracklets for the element is greater than a first time threshold, adjusting the element to a second value greater than or equal to 1.

[0150] 12. The method according to Appendix 7, wherein based on information of an image block pair for an element in the third cosine distance matrix, adjusting the element comprises:

• • [0151] adjusting a diagonal element of the third cosine distance matrix to a third value greater than or equal to 1.

[0152] 13. The method according to Appendix 7, wherein based on a clustering result of the hierarchical clustering, merging a plurality of updated tracklets belonging to the same hierarchical clustering classification comprises:

• • [0153] connecting a plurality of updated tracklets belonging to the same hierarchical clustering classification, in a time order, as a merged tracklet;
  • [0154] determining whether a case where there are a plurality of image blocks for the same time exists in an image block sequence of the merged tracklet; and
  • [0155] in a case where a determination result is “yes”, performing the following operation to update the merged tracklet;
  • [0156] removing, from the image block sequence, other image blocks than an image block with a highest detection score among the plurality of image blocks.

[0157] 14. A device for improving multi-object tracking, characterized by comprising:

• • [0158] a memory, having instructions stored thereon; and
  • [0159] at least one processor coupled to the memory and configured to execute the instructions to perform a split operation on each of a plurality of tracklets, for corresponding objects respectively, provided by a multi-object tracking model;
  • [0160] wherein the split operation comprises:
    • [0161] determining an appearance feature sequence corresponding to an image block sequence of the tracklet;
    • [0162] determining a clustering label set by clustering a plurality of appearance features in the appearance feature sequence;
    • [0163] determining, based on the clustering label set, an image block label sequence composed of clustering labels of appearance features of image blocks in the image block sequence;
    • [0164] determining a fragment label sequence corresponding to continuous fragments, having the same clustering labels, in the image block label sequence, wherein each of fragment labels in the fragment label sequence is the same clustering label of a corresponding continuous fragment;
    • [0165] in a case where a length of the fragment label sequence is greater than the number of types of the clustering labels in the clustering label set, updating the image block label sequence and the fragment label sequence by performing an update operation; and
    • [0166] splitting the tracklet based on the updated image block label sequence;
  • [0167] wherein the update operation comprises:
    • [0168] updating the fragment label sequence based on a current continuous fragment corresponding to a current fragment label in the fragment label sequence, a previous continuous fragment corresponding to a previous fragment label previous to the current fragment label, and a next continuous fragment corresponding to a next fragment label following the current fragment label; and
    • [0169] updating the image block label sequence based on the updated fragment label sequence.

[0170] 15. The device according to Appendix 14, wherein the update operation comprises:

• • [0171] in a case where the previous fragment label is same as the next fragment label,
    • [0172] if a length of the current continuous fragment is less than or equal to a first length threshold, updating the current fragment label to the previous fragment label, and updating all clustering labels in the current continuous fragment to the previous fragment label; and
    • [0173] otherwise, updating the previous fragment label to a new clustering label, updating all clustering labels in the previous continuous fragment to the new clustering label, and updating the clustering label set by adding the new clustering label into the clustering label set.

[0174] 16. The device according to Appendix 15, wherein the update operation comprises:

• • [0175] in a case where the previous fragment label is different from the next fragment label, keeping image block labels of the previous continuous fragment, the current continuous fragment, and the next continuous fragment unchanged.

[0176] 17. The device according to Appendix 15, wherein the update operation comprises:

• • [0177] in a case of selecting each of fragment labels from a second fragment label to a second to last fragment label in the fragment label sequence as the current fragment label respectively, determining whether to update the fragment label sequence.

[0178] 18. The device according to Appendix 17, wherein the update operation comprises:

• • [0179] after the current fragment label traverses each of the fragment labels from the second fragment label to the second to last fragment label in the fragment label sequence, in a case where it is determined that no fragment label update operation occurs during the traversing operation,
    • [0180] replacing a fragment label, in the fragment label sequence, that appears for a second time relative to a previous fragment label sequence with a new fragment label.

[0181] 19. The device according to Appendix 14, wherein the at least one processor is further configured to execute the instructions to: perform a merge operation on a plurality of updated tracklets obtained by performing the splitting operation on the plurality of tracklets;

• • [0182] wherein the merge operation comprises:
    • [0183] determining a first cosine distance matrix of the plurality of updated tracklets based on a first type of appearance features of respective updated tracklets associated with image block detection scores;
    • [0184] determining a second cosine distance matrix of the plurality of updated tracklets based on a second type of appearance features of respective updated tracklets associated with most similar image block pairs between tracklets;
    • [0185] determining a Hadamard product of the first cosine distance matrix and the second cosine distance matrix as a third cosine distance matrix;
    • [0186] based on information of an image block pair for an element in the third cosine distance matrix, adjusting the element;
    • [0187] performing hierarchical clustering on the plurality of updated tracklets based on the adjusted third cosine distance matrix; and
    • [0188] based on a clustering result of the hierarchical clustering, merging a plurality of updated tracklets belonging to the same hierarchical clustering classification.

[0189] 20. A computer-readable non-transitory storage medium storing a program thereon, characterized in that the program, when executed by a computer, causes the computer to perform a split operation on each of a plurality of tracklets provided for corresponding objects respectively by a multi-object tracking model;

• • [0190] wherein the split operation comprises:
    • [0191] determining an appearance feature sequence corresponding to an image block sequence of the tracklet;
    • [0192] determining a clustering label set by clustering a plurality of appearance features in the appearance feature sequence;
    • [0193] determining, based on the clustering label set, an image block label sequence composed of clustering labels of appearance features of image blocks in the image block sequence;
    • [0194] determining a fragment label sequence corresponding to continuous fragments, having the same clustering labels, in the image block label sequence, wherein each of fragment labels in the fragment label sequence is the same clustering label of a corresponding continuous fragment;
    • [0195] in a case where a length of the fragment label sequence is greater than the number of types of the clustering labels in the clustering label set, updating the image block label sequence and the fragment label sequence by performing an update operation; and
    • [0196] splitting the tracklet based on the updated image block label sequence;
  • [0197] wherein the update operation comprises:
    • [0198] updating the fragment label sequence based on a current continuous fragment corresponding to a current fragment label in the fragment label sequence, a previous continuous fragment corresponding to a previous fragment label previous to the current fragment label, and a next continuous fragment corresponding to a next fragment label following the current fragment label; and
    • [0199] updating the image block label sequence based on the updated fragment label sequence.