Method for acquiring target model, method for determining prognosis evaluation value, apparatuses, device, and medium

[0001] This application claims priority to Chinese Patent Application No. 202211186768.3, entitled “METHOD FOR ACQUIRING TARGET MODEL, METHOD FOR DETERMINING PROGNOSIS EVALUATION VALUE, APPARATUSES, DEVICE, AND MEDIUM”, filed to China National Intellectual Property Administration on Sep. 27, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the technical field of information processing, and particularly to a method for acquiring a target model, a method for determining a prognosis evaluation value, an apparatus, a device, and a medium.

[0003] Brain gliomas originate from neuroglial cells and are the most common tumors of the central nervous system, accounting for approximately 50%˜60% of intracranial tumors. The incidence shows an increasing trend annually. Patients with brain gliomas typically require prognosis evaluation after the surgery.

[0004] In the prognosis evaluation, the traditional prognosis is based on predictive factors such as the size and range of the lesion site, which has certain limitations.

[0005] The present disclosure provides a method for acquiring a target model, including:

• • [0006] acquiring a plurality of sample groups, wherein each of the sample groups corresponds to a sample user, and includes sample information of multiple modalities, and the sample information of the multiple modalities includes at least two of a magnetic resonance (MR) sample image, clinical sample information, and gene sample information; and
  • [0007] obtaining a target model by performing iterative trainings on a preset model based on the plurality of sample groups, the target model is used for predicting a prognosis evaluation value of a target object,
  • [0008] each of the iterative trainings includes:
  • [0009] performing, by using the preset model, feature extraction on the sample information of the multiple modalities in a current sample group respectively, and determining a predicted prognosis evaluation value and a consistency expression value based on extracted sample features, wherein the consistency expression value is used for characterizing a consistency degree of the sample features corresponding to a same target disease; and
  • [0010] updating parameters of the preset model based on the predicted prognosis evaluation value, a prognosis evaluation label corresponding to the current sample group, and the consistency expression value.

[0011] In an alternative embodiment, the updating parameters of the preset model based on the predicted prognosis evaluation value, a prognosis evaluation label corresponding to the current sample group, and the consistency expression value includes:

• • [0012] acquiring a difference between the predicted prognosis evaluation value and the prognosis evaluation label; and
  • [0013] updating the parameters of the preset model with a target of minimizing the difference and maximizing the consistency expression value.

[0014] In an alternative embodiment, the updating the parameters of the preset model with a target of minimizing the difference and maximizing the consistency expression value includes:

• • [0015] constructing a loss function based on the difference and the consistency expression value as follows:

[0016] $\mathrm{loss}={\sum }_i{\left(y_i^{\prime }-y_i\right)}^2-\mathrm{consistency};$

• • [0017] updating, based on a loss value of the loss function, the parameters of the preset model with the target of minimizing the difference and maximizing the consistency expression value,
  • [0018] the ‘loss’ represents the loss value, y′_i represents the predicted prognosis evaluation value, y_i represents the prognosis evaluation label, and ‘consistency’ represents the consistency expression value.

[0019] In an alternative embodiment, the determining the consistency expression value based on extracted sample features includes:

• • [0020] transposing each of the sample features, and obtaining transposed features corresponding to various sample features;
  • [0021] for two different sample features, fusing one of the two sample features with the transposed feature corresponding to the other sample feature, and obtaining a corresponding fused feature value; and
  • [0022] determining the consistency expression value based on each fused feature value.

[0023] In an alternative embodiment, the sample information of each modality includes a plurality of sub-sample information, and the performing, by using the preset model, feature extraction on the sample information of the multiple modalities in a current sample group includes:

• • [0024] for the sample information of each modality, performing feature extraction on the plurality of sub-sample information in the sample information of the modality, and obtaining a plurality of sub-feature vectors corresponding to the plurality of sub-sample information; and
  • [0025] obtaining a sample feature corresponding to the sample information of the modality by performing feature fusion on the plurality of sub-feature vectors corresponding to the sample information of each modality.

[0026] In an alternative embodiment, the obtaining a sample feature corresponding to the sample information of the modality by performing feature fusion on the plurality of sub-feature vectors corresponding to the sample information of each modality includes:

• • [0027] determining an attention value between every two sub-feature vectors, the attention value is used for characterizing a degree of closeness between the two sub-feature vectors; and
  • [0028] obtaining the sample feature corresponding to the sample information of the modality by fusing the plurality of sub-feature vectors based on the attention value.

[0029] In an alternative embodiment, the obtaining the sample feature corresponding to the sample information of the modality by fusing the plurality of sub-feature vectors based on the attention value includes:

• • [0030] for each sub-feature vector, fusing all the other sub-feature vectors into the sub-feature vector based on attention values between the sub-feature vector and all the other sub-feature vectors, and obtaining a fused sub-vector for the sub-feature vector; and
  • [0031] obtaining the sample feature corresponding to the sample information of the modality by performing re-fusion on a plurality of fused sub-vectors.

[0032] In an alternative embodiment, the parameters of the preset model include a first parameter matrix, the MR sample image includes a plurality of slice sample images, and the determining an attention value between every two sub-feature vectors includes: for each slice sample image included in the MR sample image, determining an

• • [0033] attention value between sub-feature vectors corresponding to every two slice sample images based on a current value of the first parameter matrix.

[0034] In an alternative embodiment, the parameters of the preset model includes a second parameter matrix and a third parameter matrix, and the obtaining a sample feature corresponding to the sample information of the modality by performing feature fusion on the plurality of sub-feature vectors corresponding to the sample information of each modality includes:

• • [0035] for each piece of clinical sub-sample information in the clinical sample information, performing feature fusion on sub-feature vectors corresponding to the clinical sub-sample information based on a current value of the second parameter matrix, and obtaining a sample feature corresponding to the clinical sample information; and
  • [0036] for each piece of gene sub-sample information in the gene sample information, performing feature fusion on sub-feature vectors corresponding to the gene sub-sample information based on a current value of the third parameter matrix, and obtaining a sample feature corresponding to the gene sample information.

[0037] In an alternative embodiment, the parameters of the preset model include a parameter set corresponding to the clinical sample information, the clinical sample information includes numerical sub-sample information and non-numerical sub-sample information, and the performing, by using the preset model, feature extraction on the sample information of the multiple modalities in a current sample group includes:

• • [0038] for the clinical sample information in the current sample group, converting the non-numerical sub-sample information into a first sub-feature vector;
  • [0039] obtaining a corresponding second sub-feature vector by mapping, based on a current value of each parameter in the parameter set, the numerical sub-sample information to a target space, the parameters in the parameter set are used for determining a dimension of the target space and a value at each spatial point; and
  • [0040] obtaining a sample feature corresponding to the clinical sample information by fusing the first sub-feature vector and the second sub-vector.

[0041] In an alternative embodiment, the parameter set includes a first parameter vector, a second parameter vector, and a fourth parameter matrix, and the obtaining a corresponding second sub-feature vector by mapping, based on a current value of each parameter in the parameter set, the numerical sub-sample information to a target space includes:

• • [0042] mapping the numerical sub-sample information to a first dimension in the target space based on a current value of the first parameter vector to obtain a mapping value of the first dimension, the first parameter vector is used for determining a value of a spatial point of the target space in the first dimension; and
  • [0043] determining the second sub-feature vector based on the mapping value of the first dimension, a current value of the second parameter vector, and a current value of the fourth parameter matrix, wherein the second parameter vector is used for determining a value of a spatial point of the target space in a second dimension, and the fourth parameter matrix is used for assigning a parameter to each spatial position in the first dimension and the second dimension.

[0044] In an alternative embodiment, the parameter set further includes a plurality of third parameter vectors, and after determining the second sub-feature vector based on the mapping value of the first dimension, the current value of the second parameter vector, and the current value of the fourth parameter matrix, the method further includes:

• • [0045] correcting, based on the second sub-feature vector and the plurality of third parameter vectors, the second sub-feature vector according to a formula as follows:

[0046] $\mathrm{va}=\frac{e^{\left(s{a}^T·a_1\right)}}{{\sum }_{i=1}^3{e}^{\left({\mathrm{sa}}^T·a_i\right)}}\times a_1+\frac{e^{\left(s{a}^T·a_2\right)}}{{\sum }_{i=1}^3{e}^{\left({\mathrm{sa}}^T·a_i\right)}}\times a_2+\frac{e^{\left(s{a}^T·a_3\right)}}{{\sum }_{i=1}^3{e}^{\left({\mathrm{sa}}^T·a_i\right)}}\times a_3,$

• • [0047] va is a corrected second sub-feature vector, sa is the second sub-feature vector, and a1, a2, and a3 are the third parameter vectors; and
  • [0048] the obtaining a sample feature corresponding to the clinical sample information by fusing the first sub-feature vector and the second sub-vector includes:
  • [0049] obtaining the sample feature corresponding to the clinical sample information by fusing the first sub-feature vector and the corrected second sub-vector.

[0050] In an alternative embodiment, the parameters of the preset model include a dimension parameter matrix corresponding to sample information of each modality, and the determining a predicted prognosis evaluation value and a consistency expression value based on extracted sample features includes:

• • [0051] obtaining a converted sample feature by performing, based on the sample feature and the dimension parameter matrix corresponding to the sample information of each modality, dimension transformation on the sample feature corresponding to the sample information of the modality; and
  • [0052] determining the predicted prognosis evaluation value and the consistency expression value based on converted sample features corresponding to each piece of the sample information of the multiple modalities.

[0053] In an alternative embodiment, the preset model includes a fusion module, a prediction branch, a consistency expression branch, and data processing modules corresponding to sample information of the modalities, and the performing, by using the preset model, feature extraction on the sample information of the multiple modalities in a current sample group, and determining a predicted prognosis evaluation value and a consistency expression value based on extracted sample features includes:

• • [0054] for the sample information of each modality in the current sample group, performing feature extraction on corresponding sample information using a data processing module corresponding to the sample information of the modality;
  • [0055] fusing sample features output by the data processing modules using the fusion module;
  • [0056] determining, using the prediction branch, the predicted prognosis evaluation value corresponding to a feature output by the fusion module; and
  • [0057] determining, using the consistency expression branch, the consistency expression value corresponding to the sample features output by the data processing modules.

[0058] An embodiment of the present disclosure provides a method for determining a prognosis evaluation value, including:

• • [0059] acquiring information of multiple modalities of an object to be measured, the information of the multiple modalities includes a magnetic resonance (MR) image, clinical information, and gene information; and
  • [0060] obtaining a prognosis evaluation value of the object to be measured by inputting the information of the multiple modalities into a target model,
  • [0061] the target model is obtained according to the method for acquiring a target model.

[0062] The above description is merely a summary of the technical solution of the present disclosure, which may be implemented in accordance with the contents of the specification in order to make the technical means of the present disclosure more clearly understood, and in order to make the above and other objects, features, and advantages of the present disclosure more apparent, specific implementations of the present disclosure are set forth below.

[0072] In order to make objects, solutions and advantages of embodiments of the present disclosure clearer, a clear and thorough description for technical solutions in the embodiments of the present disclosure will be given below in conjunction with the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are a part of embodiments of the present disclosure, not all the embodiments. All other embodiments obtained, based on the embodiments in the present disclosure, by those skilled in the art without paying creative effort fall within the protection scope of the present disclosure.

[0073] In the related art, the prognosis of a disease is generally evaluated based on the size of a lesion and the presence or absence of enhancement, which has certain limitations. Taking brain glioma as an example, it originates from neuroglial cells and is the most common tumor of the central nervous system, accounting for approximately 50%˜60% of intracranial tumors. The incidence shows a rising trend annually. The World Health Organization classifies gliomas into low (I and II) and high (III and IV) grades. The high-grade glioma (HGG) has a poor prognosis. Grade IV glioblastoma is the most malignant, with a 10-year survival rate of less than 3% and a median survival of about 12 to 14 months. Previous studies have limitations in evaluating the prognosis of glioma by using tumor location, size, resection range, and traditional imaging methods as prognostic predictors.

[0074] However, when using multi-factor analysis to evaluate, researchers need to follow up with many patients during diagnosis, treatment, and postoperative processes, collect diagnostic and treatment information, as well as patients' physiological index information, and then analyze and screen image prognostic factors. This work requires significant manpower and material resources. It is time-consuming and labor-intensive, and manually analyzing and screening the prognostic factors alone can take a long time, leading to low efficiency.

[0075] In view of this, the present disclosure proposes a method for prognosis evaluation, and the specific concept is as follows. Sample information of multiple modalities is taken as a training sample and trained to obtain a target model, and prognosis evaluation is performed using the target model. The sample information of the multiple modalities includes at least two of an MR sample image, clinical sample information, and gene sample information. Thus, information sources may be enriched so that prognostic factors may be screened from multiple dimensions. In the model training process, parameters of a preset model are updated with a consistency expression value as a factor so that a prognostic factor closely related to a target disease in the information of the multiple modalities may be screened to improve the clinical importance of the prognostic factor, and thus a result output by the target model has a high prognostic reference value.

[0076] Referring to FIG. 1, a flowchart of steps of a method for acquiring a target model of the present disclosure is shown. As shown in FIG. 1, the following steps may be specifically included.

[0077] At S101, a plurality sample groups are acquired, the sample groups each correspond to a plurality of sample users. The sample group includes sample information of multiple modalities, and the sample information of the multiple modalities includes at least two of an MR (magnetic resonance) sample image, clinical sample information, and gene sample information.

[0078] At S102, a target model is obtained by performing iterative trainings on a preset model based on the plurality of sample groups. The target model is configured to predict a prognosis evaluation value of a target object.

[0079] The following steps are performed in each iterative training in S102.

[0080] At S1021, a feature extraction is performed on the sample information of the multiple modalities in a current sample group using the preset model, and a predicted prognosis evaluation value and a consistency expression value are determined based on extracted sample features. The consistency expression value is used for characterizing a consistency degree of the sample information of the multiple modalities reflecting a same target disease.

[0081] At S1022, parameters of the preset model are updated based on the predicted prognosis evaluation value, a prognosis evaluation label corresponding to the current sample group, and the consistency expression value.

[0082] In the embodiment, the sample user may refer to a user who clearly knows the prognostic survival time and quality of life for suffering from a target disease. The target disease may be any clinically known disease, such as intracranial tumor as mentioned above, common hepatobiliary and pulmonary tumors, which is not specifically limited here.

[0083] The age and gender of the sample users may be diversified as much as possible. The sample information of the multiple modalities may include at least two of the MR sample image, the clinical sample information, and the gene sample information. The sample information of different modalities may be used for reflecting a characterization in a corresponding dimension after the sample user suffers from the target disease. For example, MR sample information may reflect a morphological characterization of an organ tissue after the sample user suffers from the target disease, the clinical sample information may reflect a characterization of a diagnostic treatment after the sample user suffers from the target disease, and the gene sample information may reflect a characterization of an expression state of certain genes after the sample user suffers from the target disease.

[0084] The MR sample image may be used for reflecting a morphological feature of a corresponding organ tissue after the sample user suffers from the target disease. Specifically, the MR sample image may be directly acquired from a magnetic resonance imaging (MRI) device, a memory, or any other suitable source. For example, the MR sample image of the sample user may be acquired from a medical database after obtaining the corresponding authority. It should be noted that the MR sample image is an image related to the target disease of the sample user. For example, if the sample user suffers from a brain glioma, the MR sample image is an MR image of the brain of the sample user, which may reflect morphological features of brain tissues. As another example, if the sample user suffers from a liver tumor, the MR sample image may be an MR image of the abdomen of the user, which may reflect morphological features of abdominal tissues.

[0085] The clinical sample information may include information of the sample user during diagnosis and therapy, such as drug information, hospitalization information, therapeutic regimen information, attending physician information, and hospital information.

[0086] The gene sample information may include information of genes related to the occurrence and prognosis of the target disease. Specifically, information of each gene in the gene sample information includes a name of the gene and an expression state of the gene, because the onset and prognosis of the disease may be reflected in the expression of some genes. For example, in the case of brain glioma, taking the telomerase reverse tranase (TERT) gene as an example, it is one of the important genes encoding a telomerase complex, the TERT gene has no transcriptional activity in most non-tumor cells, but TERT gene mutation such as promoter mutation, gene translocation, and DNA amplification exists in 73% of tumors. That is, an expression category of the above-mentioned genes has a certain correlation with tumors.

[0087] In the embodiment, at least two of the MR sample image, the clinical sample information, and the gene sample information may be taken as a sample group of the sample user. In a specific example, the MR sample image in the sample group may be necessary, namely, any one or both of the clinical sample information and the gene sample information may be combined with the MR sample image to obtain the sample group. Specifically, the sample group may include the MR sample image and the clinical sample information, or may include the MR sample image and the gene sample information, or may include the MR sample image, the clinical sample information, and the gene sample information.

[0088] The preset model may be subjected to iterative trainings with a plurality of sample groups as the training samples. Specifically, during each training, a plurality of sample groups may be input into the preset model in batches, or only one sample group is input into the preset model.

[0089] In the embodiment, in each iterative training, the feature extraction may be performed on the sample information of the multiple modalities in the current sample group using the preset model. The current sample group refers to a sample group input into the preset model at the current time, and in the case of inputting a plurality of sample groups to the preset model at a time, the current sample group refers to any sample group among the plurality of sample groups input at the current time.

[0090] The preset model may be configured to perform a feature extraction on the sample information of each modality in the current sample group to obtain the sample feature corresponding to each sample information. Specifically, for the MR sample image, the extracted feature is a radiomics feature vector. For the clinical sample information, the extracted feature is a feature vector obtained through performing feature vector conversion on the clinical information. For the gene sample information, the extracted feature is a feature vector obtained through performing feature vector conversion on the gene information.

[0091] The predicted prognosis evaluation value of the sample object corresponding to the current sample group may be determined according to the sample feature extracted from information of each modality. In a specific implementation, a feature fusion may be performed on the sample features corresponding to information of a plurality of modality, then the predicted prognosis evaluation value is determined based on a fused feature.

[0092] In practice, the sample information of different modalities should all contain a description of the target disease, and each should contain a large amount of consistent information about the target disease. However, the sample information of different modalities also contains information unrelated to the target disease. For example, the MR sample image contains images of other normal tissue sites in addition to the lesion sites, and the images of normal tissue sites may not have related information in the sample information of other modalities, for example, this part of information is not contained in the clinical sample information, so the preset model should learn to discard this part of useless information in the training process.

[0093] Therefore, in the embodiment, the preset model may determine the consistency expression value based on the sample features corresponding to the information of the plurality of modality, and the consistency expression value is incorporated into the construction of a loss function to update the parameters of the preset model, so that the preset model continuously enhances an expression degree of the extracted sample features on the target disease based on the consistency expression value.

[0094] In this way, the consistency expression value may be used for characterizing the consistency degree of the sample features corresponding to the information of the plurality of modality reflecting the target disease, namely, whether the extracted sample features consistently express the target disease.

[0095] Illustratively, when the consistency expression value is high, it means that the sample features extracted from the MR sample image, the clinical sample information, and the gene sample information are all features for expressing the target disease. For the MR sample image, the extracted sample features are features for expressing the lesion site of the target disease. For the clinical sample information, the extracted sample features are the expression of the diagnosis and the therapeutic regimen of the target disease, and the expression of the target disease in the patient's age, occupation, etc. For the gene sample information, the extracted sample features are used for characterizing the expression of the target disease on related genes.

[0096] The prognosis evaluation label of the present disclosure is a real prognosis condition of the sample object. If the prognostic survival time needs to be predicted, the prognosis evaluation label is a real prognostic survival time of the sample object, and the predicted prognosis evaluation value may be represented as a prognostic age. If the prognostic quality of life needs to be predicted, the prognosis evaluation label is a real prognostic quality of life grade of the sample object, including high grade, low grade, and medium grade, and the predicted prognosis evaluation value may be represented as the prognostic quality of life grade.

[0097] By adopting the technical solution of the embodiment, when the parameters of the preset model are updated, the consistency expression value, the predicted prognosis evaluation value, and the prognosis evaluation label are incorporated in the construction of the loss function. Thus, with the training of the model, on the one hand, the expression degree of the extracted sample features on the target disease may be enhanced based on the consistency expression value, so that the prognostic factor for determining the prognosis evaluation value is strongly related to the target disease. Therefore, the clinical importance of the prognostic factor may be improved so that the prognosis evaluation value can be more medically referenced. On the other hand, the parameters of the preset model may be continuously updated based on a difference between the predicted prognosis evaluation value and the prognosis evaluation label. Therefore, the predicted prognosis evaluation value predicted through the model may approach the prognosis evaluation label infinitely with the proceeding of the training, and then the prognosis evaluation value may approach the prognosis evaluation value under the real situation. Thus, a clinical reference value of the prognosis evaluation value may be further improved.

[0098] On another hand, in the present disclosure, the prognosis is predicted using the sample information of the multiple modalities. Thus, the complementarity between the sample information of different modalities may be used as the prognostic factor of the target disease, thereby enriching the data source of the prognostic factors, and improving the medical reference of the prognosis evaluation value.

[0099] In an alternative implementation, in the case that the target disease is glioma, the gene sample information may include information of at least one gene of isocitrate dehydrogenase (IDH), a chromosome 1p/19q joint deletion state, a TERT gene promoter, and an O6-methylguanine-DNA methyltransferase (MGMT) promoter methylation; the clinical sample information includes at least one clinical information of gender, age, histological diagnosis, tumor grading, medication information, and history of malignant tumors.

[0100] The information of each gene in the gene sample information may include the name of the gene and the expression category of the gene, and the expression category may be different according to different genes. Specifically, the expression categories in the information of the IDH gene are mutant type and wild type. The expression states of the 1p/19q gene include a deleted state and an undeleted state. The expression categories of the MGMT promoter are methylated and unmethylated. The expression categories of the TERT promoter include mutant type and wild type, and the information of various types of genes is specifically shown in the following Table 1.

[0101]

TABLE 1
Gene sample information
Name	IDH	1p/19q	TERT	MGMT
TCGA-CS-4938	Mutant	non-codel	WT	Unmethylated
TCGA-CS-4941	WT	non-codel	Mutant	Methylated

[0102] The clinical sample information may include one or more of the gender, age, histological diagnosis, tumor grading, medication information, and history of malignant tumors of the sample object. Of course, in practical situations, the clinical sample information may not only be limited to the above-mentioned clinical information, but also include more clinical information, such as the occupation and region of the sample object, any information related to the occurrence and prognosis of the target disease may be used as the clinical information.

[0103] In an example, the clinical sample information may be as shown in Table 2 below.

[0104]

TABLE 2
Clinical sample information
							Survival
Name	Age	Gender	Histology	Grade	Drug_name	History_other_malignancy	(months)
TCGA-	31	female	astrocytoma	G2	Temodar	No	4.69
CS-4938
TCGA-	67	male	astrocytoma	G3	None	No	7.68
CS-4941

[0105] In an embodiment, the sample information of the multiple modalities may include the MR sample image, the clinical sample information, and the gene sample information. That is, each sample group includes the MR sample image, the clinical sample information, and the gene sample information. Therefore, the feature extraction is required to be performed on the sample information of three modalities in each iterative training.

[0106] In the training process, each iterative training includes stages of feature extraction and constructing the loss function based on the output of the model to update the parameters.

[0107] The sample information of each modality may include a plurality of sub-sample information. In the stage of feature extraction, feature extraction may be performed on each sub-sample information in the sample information of each modality. Then, a feature fusion is performed on features of a plurality of sub-sample information in the sample information of one modality to obtain a sample feature corresponding to the sample information of this modality.

[0108] Hereinafter, the two stages are described separately.

Stage 1: Stage of Feature Extraction.

[0109] As described above, in one implementation, the sample information of each modality includes a plurality of sub-sample information. For the sample information of each modality, feature extraction is performed on a plurality of sub-sample information in the sample information of the modality to obtain corresponding multiple sub-feature vectors. Then, a feature fusion is performed on the multiple sub-feature vectors corresponding to the sample information of each modality to obtain a sample feature corresponding to the sample information of this modality.

[0110] For the feature extraction of each sub-sample information, as described in the above-mentioned embodiment, after obtaining the sub-feature vector corresponding to the sub-sample information, a feature fusion may be performed according to the following process to obtain the sample feature.

[0111] For the MR sample image, the sub-sample information may be one slice image of the MR sample image, and when feature extraction is performed, feature extraction may be performed on each slice image to obtain sub-feature vectors corresponding to various slice images. Then, the sub-feature vectors corresponding to the slice images are fused to obtain a sample feature of the MR sample image.

[0112] For the clinical sample information, the sub-sample information may be a kind of clinical information in the clinical sample information. For example, the age, gender, and tumor grading in the clinical information may all be taken as a piece of sub-sample information. When feature extraction is performed, feature extraction is performed on each kind of information. Specifically, each kind of information is converted into a feature vector to obtain the sub-feature vector corresponding to this kind of information, and then the sub-feature vectors corresponding to various kinds of clinical information in the clinical sample information are fused to obtain a sample feature of the clinical sample information.

[0113] For the gene sample information, the sub-sample information may be information of one gene in the gene sample information, such as information of IDH and information of the TERT gene promoter. Then, the information of each gene may be converted into the sub-feature vector, and then the sub-feature vectors corresponding to the information of various genes in the gene sample information are fused to obtain a sample feature of the gene sample information.

[0114] In an alternative example, when the sub-feature vectors corresponding to the sample information of each modality are fused, the fusion may be performed directly according to a preset weight corresponding to each sub-feature vector. For example, a preset weight may be manually set for each slice image in advance, a weight may also be set for each kind of clinical information, and a corresponding preset weight may also be set for information of each gene. The preset weight may characterize the importance of this kind of sub-sample information to the prognosis. Thus, feature fusion may be performed based on the importance of each sub-feature vector to obtain features that are important to prognosis evaluation, thereby improving the medical value of prognosis evaluation.

[0115] In yet another example, an attention mechanism may be fused in the preset model. For the sample information of each modality, a correlation degree between the sub-feature vectors may be determined through the attention mechanism, and then the sub-feature vectors are fused based on the correlation degree, so that the features with a higher correlation degree in the sample information of the modality may be fused to improve the correlation between the prognostic factors, thereby making the prognostic factors for the prognosis evaluation of the target model closely related, further improving the clinical importance, and making the target model have medical interpretability.

[0116] In a specific implementation, an attention value between every two sub-feature vectors may be determined. The plurality of sub-feature vectors are fused based on the attention value to obtain the sample feature corresponding to the sample information of the modality.

[0117] As described above, the attention value is used for characterizing a degree of closeness between the two sub-feature vectors.

[0118] When the sub-feature vectors are fused based on the attention value between every two sub-feature vectors, in one example, the two sub-feature vectors may be fused based on the attention value between every two sub-feature vectors to obtain a fused vector, thereby obtaining a plurality of fused vectors. Then, the plurality of fused vectors are fused to obtain the sample feature. For example, the sub-feature vectors include a vector i, a vector j, and a vector k, and when being fused, the vector i and the vector j may be fused to obtain a fused vector ij. Similarly, a fused vector ik and a fused vector jk are obtained, and then the fused vector ij, the fused vector ik, and the fused vector jk are fused to obtain the sample feature.

[0119] In yet another example, for each sub-feature vector, all the other sub-feature vectors are fused into the sub-feature vector based on attention values between the sub-feature vector and all the other sub-feature vectors to obtain a fused sub-vector for the sub-feature vector. Re-fusion is performed on a plurality of fused sub-vectors to obtain the sample feature corresponding to the sample information of the modality.

[0120] Illustratively, the sub-feature vectors include the vector i, the vector j, and the vector k, and when being fused, the vector j and the vector k may be fused into the vector i according to the attention value between the vector i and the vector j and the attention value between the vector i and the vector k to obtain a fused sub-vector i′. Similarly, a fused sub-vector j′ and a fused sub-vector k′ are obtained, and then the fused sub-vector i′, the fused sub-vector j′, and the fused sub-vector k′ are fused to obtain the sample feature.

[0121] For the clinical sample information, the attention value may be determined according to the following process, and feature fusion may be performed based on the attention value.

[0122] The attention value between every two sub-feature vectors may be determined based on an average attention value of the plurality of clinical information and the two sub-feature vectors. The average attention value of the plurality of clinical information may be acquired by referring to the following formula (1).

[0123] $\begin{array}{cc}S=e^{\left({\mathrm{va}}^T·\mathrm{vg}\right)}+e^{\left({\mathrm{va}}^T·\mathrm{vh}\right)}+e^{\left({\mathrm{va}}^T·\mathrm{vgr}\right)}+e^{\left({\mathrm{va}}^T·\mathrm{vd}\right)}+e^{\left({\mathrm{va}}^T·\mathrm{vhom}\right)}& \mathrm{Formula}\left(1\right)\end{array}$

[0124] va represents a sub-feature vector corresponding to age, vg represents a sub-feature vector corresponding to gender, vh represents a sub-feature vector corresponding to histological diagnosis, vhom represents a sub-feature vector corresponding to history of malignant tumors, vd represents a sub-feature vector corresponding to medication information, and vgr represents a sub-feature vector corresponding to tumor grading information. S represents the average attention value.

[0125] Then, the attention value between every two sub-feature vectors may be determined according to the following formula (2):

[0126] $\begin{array}{cc}\mathrm{Si}=\frac{e\left({\mathrm{vi}}^T·\mathrm{vj}\right)}{S}& \mathrm{Formula}\left(2\right)\end{array}$

[0127] Si represents an attention value between a sub-feature vector vi and a sub-feature vector vj.

[0128] Accordingly, when feature fusion is performed, one sub-feature vector may be fused with all the other sub-feature vectors, and then the obtained fused sub-vectors are fused. Specifically, taking the fusion of the sub-feature vector corresponding to the age information in the clinical information as an example, a fused sub-vector corresponding to the age information may be determined according to the following formula (3):

[0129] $\begin{array}{cc}\mathrm{va\_att}=\frac{e^{\left(v{a}^T·\mathrm{vg}\right)}}{S}\times \mathrm{vg}+\frac{e^{\left(v{a}^T·\mathrm{vh}\right)}}{S}\times \mathrm{vh}+\frac{e^{\left(v{a}^T·\mathrm{vgr}\right)}}{S}\times \mathrm{vgr}+\frac{e^{\left(v{a}^T·\mathrm{vd}\right)}}{S}\times \mathrm{vd}+\frac{e^{\left(v{a}^T·\mathrm{vhom}\right)}}{S}\times \mathrm{vhom}& \mathrm{Formula}\left(3\right)\end{array}$

[0130] In formula (3), va_att is the fused sub-vector corresponding to the age information.

[0131] For the gene sample information, processes of determining the attention value and performing feature fusion based on the attention value may be performed by referring to the above-mentioned clinical sample information, and will not be described in detail herein.

[0132] For the MR sample image, the attention value may be determined according to the following process, and feature fusion may be performed based on the attention value.

[0133] In an example, in the stages of feature extraction and feature fusion, since it is necessary to extract a prognostic factor of higher clinical importance, namely, to extract a feature vector of higher clinical importance, in the example, parameter matrices may be set for the stages of feature extraction and feature fusion, and the parameter matrix may be continuously updated with the training of the model to extract the prognostic factor of higher importance.

[0134] Specifically, as described above, the MR sample image includes a plurality of slice sample images, each slice sample image being a piece of sub-sample information. Specifically, for each slice sample image included in the MR sample image, the attention value between the sub-feature vectors corresponding to every two slice sample images is determined based on a current value of a first parameter matrix. Then, the sub-feature vectors may be fused based on the attention value.

[0135] In a specific implementation, the attention value between the sub-feature vectors corresponding to every two slice sample images may be determined according to the following formula (4):

[0136] $\begin{array}{cc}{\alpha }_{i,j}=\frac{e^{{\left(Q\times V_i\right)}^T·\left(K\times V_j\right)}}{{\sum }_j{e}^{{\left(Q\times V_i\right)}^T·\left(K\times V_j\right)}}& \mathrm{Formula}\left(4\right)\end{array}$

[0137] In formula (4), Q and K are the first parameter matrices, where values of Q and K may be different. In practice, Q and K may be a parameter matrix of 512×512. v_i is a sub-feature vector corresponding to an i-th slice sample image. v_j is a sub-feature vector corresponding to a j-th slice sample image. α_i,j is an attention value between the i-th slice sample image and the j-th slice sample image.

[0138] Then, for each sub-feature vector, all the other sub-feature vectors are fused into the sub-feature vector based on attention values between the sub-feature vector and all the other sub-feature vectors to obtain the fused sub-vector for the sub-feature vector according to the following formula (5):

[0139] $\begin{array}{cc}S{V}_i^1={\sum }_{1=1}^n{\alpha }_{i,j}{V}_j^1& \mathrm{Formula}\left(5\right)\end{array}$

[0140] In formula (5),

[0141] $S{V}_i^1$
is a fused sub-vector corresponding to the i-th slice sample image, and n represents a total number of the slice sample images.

[0142] Later, the fused sub-vectors are fused to obtain the sample feature of the MR sample image according to the following formula (6):

[0143] $\begin{array}{cc}SV=\frac{{\Sigma }_{i=1}^{n}S{V}_i}{n}& \mathrm{Formula}\left(6\right)\end{array}$

[0144] SV represents the sample feature of the MR sample image, and SV_i represents the fused sub-vector corresponding to the i-th slice sample image.

[0145] In a further embodiment, since the parameter matrices are set for the stages of feature extraction and feature fusion, and the parameter matrix may be continuously updated with the training of the model, the prognostic factor of higher importance is extracted. The clinical sample information and the gene sample information may be further fused based on the set parameter matrix when the sub-feature vectors are fused.

[0146] For each piece of clinical sub-sample information in the clinical sample information, feature fusion is performed on sub-feature vectors corresponding to the clinical sub-sample information based on a current value of a second parameter matrix to obtain a sample feature corresponding to the clinical sample information.

[0147] For each piece of gene sub-sample information in the gene sample information, feature fusion is performed on sub-feature vectors corresponding to the gene sub-sample information based on a current value of a third parameter matrix to obtain a sample feature corresponding to the gene sample information.

[0148] Specifically, after obtaining each fused sub-vector corresponding to the sub-feature vector from the above-mentioned formula (3), feature fusion may be performed on fused sub-vectors corresponding to the sub-feature vectors corresponding to the clinical sub-sample information based on the current value of the second parameter matrix to obtain the sample feature corresponding to the clinical sample information. Specifically, fusion may be performed according to the following formula (7) and formula (8):

[0149] $\begin{array}{cc}\mathrm{CV}=\frac{v{p}^T·\mathrm{va\_att}}{S_1}\times \mathrm{va\_att}+\frac{v{p}^T·\mathrm{vg\_att}}{S_1}\times \mathrm{vg\_att}+\frac{v{p}^T·\mathrm{vh\_att}}{S_1}\times \mathrm{vh\_att}+\frac{v{p}^T·\mathrm{vgr\_att}}{S_1}\times \mathrm{vgr\_att}+\frac{v{p}^T·\mathrm{vd\_att}}{S_1}\times \mathrm{vd\_att}+\frac{v{p}^T·\mathrm{vhom\_att}}{S_1}\times \mathrm{vhom\_att};& \mathrm{Formula}\left(7\right)\end{array}$ $\begin{array}{cc}{S}_1={\mathrm{vp}}^T·\left(\mathrm{ya\_att}+\mathrm{yg\_att}+\mathrm{vh\_att}+\mathrm{vgr\_att}+\mathrm{vd\_att}+\mathrm{vhom\_att}\right);& \mathrm{Formula}\left(8\right)\end{array}$

[0150] CV represents the sample feature corresponding to the clinical sample information, and vp is the second parameter matrix, which may be a parameter vector of 128×1.

[0151] The fusion of fused sub-vectors corresponding to the gene sub-sample information may also be performed by referring to the above-mentioned formula (7) and formula (8). The second parameter matrix and the third parameter matrix may both be parameter vectors of 128×1, and parameters in the second parameter matrix and the third parameter matrix are updated with the update of the preset model (that is, they are updated according to a loss value of the loss function).

[0152] In a further example, the clinical sample information includes numerical sub-sample information and non-numerical sub-sample information. In the stage of feature extraction, based on a parameter set that is set for the above-mentioned preset model, the numerical sub-sample information is mapped to a vector space according to the parameter set for the model, and parameters in the parameter set are continuously updated during the training process, so that numerical clinical information of different sample objects may be mapped within a spatial range, and then a comprehensive prognostic factor of the numerical clinical information may be extracted. For example, for different patients of the same age, the parameters in the parameter set are updated after a sample group of a patient A is input into the preset model for training. Later, when a sample group of a patient B is input into the preset model for training, vector mapping is performed on the age based on the updated parameters in the parameter set. Therefore, in the training process, the feature vectors corresponding to different patients of the same age may be different, but change within a certain spatial range. Thus, the influence of age group on the prognosis may be obtained in the prognosis evaluation.

[0153] In a specific implementation, for the clinical sample information, in each iterative training, for the clinical sample information in the current sample group, the non-numerical sub-sample information is converted into a first sub-feature vector. Moreover, the numerical sub-sample information is mapped to a target space based on a current value of each parameter in the parameter set to obtain a corresponding second sub-feature vector. The first sub-feature vector and the second sub-feature vector are fused to obtain the sample feature corresponding to the clinical sample information.

[0154] In the embodiment, the parameters in the parameter set are used for determining a dimension of the target space and a value at each spatial point.

[0155] The non-numerical sub-sample information may refer to sub-sample information in a character string format or sub-sample information of a literal type. For example, the gender “male” is the sub-sample information of the literal type, and the tumor grading is the sub-sample information of the character string type. For the non-numerical sub-sample information, the feature vectors corresponding to the sub-sample information may be preset in advance. Then, for the clinical sample information in the current sample group, the first sub-feature vector corresponding to the non-numerical sub-sample information in the current sample group may be obtained by looking up a table. The table includes feature vectors corresponding to various non-numerical sub-sample information, which may be understood as a fixed feature vector.

[0156] The numerical sub-sample information may be sub-sample information of a numerical type. For example, the age “62” is the sub-sample information of the numerical type. For this type of sub-sample information, the numerical sub-sample information may be mapped to the target space based on the current values of the parameters in the parameter set. The target space may be a vector space, and the vector space may be a two-dimensional space, including a plurality of values in a first dimension and a plurality of values in a second dimension on the two-dimensional space. That is, a numerical value may be distributed to various positions in the target space to obtain a second sub-feature vector corresponding to the numerical value.

[0157] In this way, parameters in the parameter set may be understood to be weight values that distribute a numerical value to various positions in the target space.

[0158] The process of fusing the first sub-feature vector and the second sub-feature vector to obtain the sample feature corresponding to the clinical sample information may be described by referring to the above-mentioned example. For example, the first sub-feature vector and the second sub-feature vector together constitute a plurality of sub-feature vectors, which may be fused according to the above-mentioned fusion method for the plurality of sub-feature vectors.

[0159] Specifically, the parameter set includes a first parameter vector, a second parameter vector, and a fourth parameter matrix, and when the numerical sub-sample information is mapped to the target space based on the current value of each parameter in the parameter set to obtain the corresponding second sub-feature vector, the numerical sub-sample information is mapped to a first dimension in the target space based on a current value of the first parameter vector to obtain a mapping value of the first dimension. The second sub-feature vector is determined based on the mapping value of the first dimension, a current value of the second parameter vector, and a current value of the fourth parameter matrix.

[0160] The first parameter vector is used for determining a value of a spatial point of the target space in the first dimension. The second parameter vector is used for determining a value of a spatial point of the target space in a second dimension. The fourth parameter matrix is used for assigning a parameter to each spatial position in the first dimension and the second dimension.

[0161] The first parameter vector may be understood to be a weight value that distributes a numerical value to various positions in the first dimension of the target space. The second parameter vector may be understood to be a weight value that distributes a numerical value to various positions in the second dimension of the target space. The fourth parameter matrix may be understood to be a weight value that distributes a numerical value to each position in the target space.

[0162] Specifically, the numerical sub-sample information may be mapped to the first dimension in the target space based on the current value of the first parameter vector to obtain the mapping value of the first dimension according to the following formula (9):

[0163] $\begin{array}{cc}\mathrm{temp\_a}=\mathrm{sigmoid}\left(w\times a\right)& \mathrm{Formula}\left(9\right)\end{array}$

[0164] In formula (9), w is the first parameter vector, which in practice may be the parameter vector of the 128× 1 dimension. a is the numerical sub-sample information, for example, a is an age value “62”.

[0165] Then, the second sub-feature vector may be determined based on the mapping value of the first dimension, the current value of the second parameter vector, and the current value of the fourth parameter matrix according to the following equation (10):

[0166] $\begin{array}{cc}\mathrm{sa}=W\times \mathrm{temp\_a}+b& \mathrm{Formula}\left(10\right)\end{array}$

[0167] In formula (10), b is the second parameter vector, which in practice may be the parameter vector of the 128×1 dimension. W is the fourth parameter matrix, which in practice may be a parameter matrix of 128. Sa is the second sub-feature vector.

[0168] Of course, in yet another embodiment, the parameter set further includes a plurality of third parameter vectors. After determining the second sub-feature vector, the second sub-feature vector may also be modified specifically according to the following formula (11):

[0169] $\begin{array}{cc}\mathrm{va}=\frac{e^{\left(s{a}^T·a_1\right)}}{{\Sigma }_{i=1}^3{e}^{\left(s{a}^T·a_i\right)}}\times a_1+\frac{e^{\left(s{a}^T·a_2\right)}}{{\Sigma }_{i=1}^3{e}^{\left(s{a}^T·a_i\right)}}\times a_2+\frac{e^{\left(s{a}^T·a_3\right)}}{{\Sigma }_{i=1}^3{e}^{\left(s{a}^T·a_i\right)}}\times a_3& \mathrm{Formula}\left(11\right)\end{array}$

[0170] va is a corrected second sub-feature vector, sa is the second sub-feature vector, and a₁, a₂, and a₃ are the third parameter vectors. Specifically, these three third parameter vectors may all be parameter vectors of 128*1, where differences are allowed between these three third parameter vectors.

[0171] Accordingly, the first sub-feature vector and the corrected second sub-feature vector may be fused to obtain the sample feature corresponding to the clinical sample information.

[0172] In an alternative example, when the sample features corresponding to the sample information of the multiple modalities are fused, the sample features may be mapped to the same space and then fused. Specifically, the parameters of the preset model include a dimension parameter matrix corresponding to the sample information of each modality. In this way, dimension transformation may be performed on the sample feature corresponding to the sample information of each modality based on the sample feature and the dimension parameter matrix corresponding to the sample information of the modality to obtain a converted sample feature. The predicted prognosis evaluation value and the consistency expression value are determined based on the converted sample feature corresponding to each piece of the sample information of the multiple modalities.

[0173] Each dimension parameter matrix is used for adjusting the dimension of the sample feature corresponding to the sample information of the modality. Specifically, dimension adjustment may be performed on the sample feature corresponding to the sample information of the modality according to the following formulas (12) to (14):

[0174] $\begin{array}{cc}{\mathrm{PV}}_1=M_1·\mathrm{SV}& \mathrm{Formula}\left(12\right)\end{array}$ $\begin{array}{cc}{\mathrm{PV}}_2=M_2·\mathrm{CV}& \mathrm{Formula}\left(13\right)\end{array}$ $\begin{array}{cc}{\mathrm{PV}}_3=M_3·\mathrm{GV}& \mathrm{Formula}\left(14\right)\end{array}$

[0175] PV₁ is a converted sample feature corresponding to the MR sample image, PV₂ is a converted sample feature corresponding to the clinical sample information, and PV₃ is a converted sample feature corresponding to the gene sample information.

[0176] SV is the sample feature corresponding to the MR sample image, CV is the sample feature corresponding to the clinical sample information, and GV is the sample feature corresponding to the gene sample information.

[0177] M₁ is a dimension parameter matrix corresponding to the MR sample image, which may be a parameter matrix of 64×512. M₂ is a dimension parameter matrix corresponding to the clinical sample information, which may be a parameter matrix of 64×128. M₃ is a dimension parameter matrix corresponding to the gene sample information, which may be the parameter matrix of 64× 128.

Stage 2: Stage of Updating Parameters.

[0178] In the present disclosure, it is necessary to update the parameters of the preset model based on the predicted prognosis evaluation value, the prognosis evaluation label corresponding to the current sample group, and the consistency expression value. In a specific implementation, the difference between the predicted prognosis evaluation value and the prognosis evaluation label may be acquired. The parameters of the preset model are updated with a target of minimizing the difference and maximizing the consistency expression value.

[0179] The difference between the predicted prognosis evaluation value and the prognosis evaluation label may reflect a distance between the prognosis evaluation value predicted through the preset model and a real prognosis evaluation value. The consistency expression value may characterize the consistency degree of the sample features corresponding to the same target disease. In the training process, the training target may be minimizing the difference between the predicted prognosis evaluation value and the prognosis evaluation label and maximizing the consistency expression value.

[0180] In this way, the parameters of the preset model may be updated with the target of minimizing the difference and maximizing the consistency expression value.

[0181] In a specific implementation, when updating the parameters of the preset model with the target of minimizing the difference and maximizing the consistency expression value, a loss function as shown in the following formula (15) may be constructed based on the difference and the consistency expression value:

[0182] $\begin{array}{cc}\mathrm{loss}={\sum }_i{\left(y_i^{\prime }-y_i\right)}^2-\mathrm{consistency}& \mathrm{Formula}\left(15\right)\end{array}$

[0183] The parameters of the preset model are updated based on a loss value of the loss function with the target of minimizing the difference and maximizing the consistency expression value. The loss represents the loss value,

[0184] $y_i^{\prime }$
represents the predicted prognosis evaluation value, y_i represents the prognosis evaluation label, and consistency represents the consistency expression value.

[0185] As can be seen from the above-mentioned loss function, the training target is to minimize the loss value, then in order that the loss value may be minimized, it is necessary to minimize the difference and maximize the consistency expression value. It is noted that the consistency expression value of the present disclosure may be a numerical value between 0 and 1.

[0186] In an example, the consistency expression value may be determined as follows.

[0187] Firstly, each of the sample features is transposed to obtain a transposed feature corresponding to each of the sample features. Then, for two different sample features, one of the two sample features is fused with the transposed feature corresponding to the other sample feature to obtain a corresponding fused feature value. The consistency expression value is determined based on each fused feature value.

[0188] In an embodiment, the transposed feature corresponding to each of the sample features may be determined based on the following formula (16):

[0189] $\begin{array}{cc}P{V}^T·\mathrm{PV}=1& \mathrm{Formula}\left(16\right)\end{array}$

[0190] In formula (16), PV is the sample feature, PV^T is the transposed feature, and a result of dot product between the two is 1. That is, the sample features may be normalized by transposing.

[0191] Then, one of the two sample features is fused with the transposed feature corresponding to the other sample feature to obtain the corresponding fused feature value according to the following formula (17):

[0192] $\begin{array}{cc}\mathrm{PV}12={\mathrm{PV}}_1^T·{\mathrm{PV}}_2& \mathrm{Formula}\left(17\right)\end{array}$

[0193] In formula (18),

[0194] $P{V}_1^T$
is a transposed feature of a sample feature 1, and PV₂ is a sample feature 2, and a result of dot product between the two is between 0 and 1.

[0195] Taking the sample information including the MR sample information, the clinical sample information, and the gene sample information as an example, the consistency expression value may be determined according to formula (18):

[0196] $\begin{array}{cc}\mathrm{consistency}={\mathrm{PV}}_1^T·{\mathrm{PV}}_2+P{V}_2^T·{\mathrm{PV}}_3+P{V}_3^T.& \mathrm{Formula}\left(18\right)\end{array}$

[0197] PV₁ is the sample feature of the MR sample image, PV₂ is the sample feature of the clinical sample information, and PV₃ is the sample feature of the gene sample information.

[0198] $P{V}_1^T$
is a transposed feature corresponding to the sample feature of the MR sample image,

[0199] $P{V}_2^T$
is a transposed feature corresponding to the sample feature of the clinical sample information, and

[0200] $P{V}_3^C$
is a transposed feature corresponding to the sample feature of the gene sample information.

[0201] Referring to FIG. 2, a schematic structural diagram of a preset model of the present disclosure is shown. As shown in FIG. 2, the preset model may include a fusion module, a prediction branch, a consistency expression branch, and a data processing module corresponding to the sample information of each modality.

[0202] When performing feature extraction on the sample information of the multiple modalities in the current sample group using the preset model, and determining the predicted prognosis evaluation value and the consistency expression value based on the extracted sample features, for the sample information of each modality in the current sample group, feature extraction is performed on corresponding sample information using the data processing module corresponding to the sample information of the modality. Sample features output by the data processing modules are fused using the fusion module. Later, a predicted prognosis evaluation value corresponding to a feature output by the fusion module is determined using the prediction branch. Then, the consistency expression value corresponding to the sample features output by the data processing modules is determined using the consistency expression branch.

[0203] That is, the sample information of each modality of the present disclosure is input into a corresponding data processing module, and the sample information of the modality is subjected to feature extraction by the corresponding data processing module. Later, the extracted sample features are input into the fusion module and fused by the fusion module. Then, a fused sample feature is input into a prediction module, and the prediction module determines the predicted prognosis evaluation value based on the fused sample feature.

[0204] Hereinafter, the method for acquiring a target model of the present disclosure is exemplarily illustrated in conjunction with the preset model shown in FIG. 2.

[0205] At S1, training samples are prepared. The training samples include sample groups corresponding to a plurality of sample users, and each sample group includes the MR sample image, the clinical sample information, and the gene sample information of the sample user.

[0206] At S2, the sample groups are input into the preset model. The MR sample image in the sample group is input into an image data processing module, the clinical sample information is input into a clinical data processing module, and the gene sample information is input into a gene data processing module to perform feature extraction of the sample information of the multiple modalities and obtain the sample feature corresponding to the sample information of each modality.

[0207] Specifically, the image data processing module is configured to perform feature extraction on a plurality of slice sub-sample images in the MR sample image. After the extraction, feature fusion is performed on sub-feature vectors corresponding to the slice sub-sample images to obtain the sample feature corresponding to the MR sample image.

[0208] Referring to FIG. 3 and FIG. 4, FIG. 3 is a schematic structural diagram of the image data processing module, and FIG. 4 is a schematic structural diagram of a ResNet network in FIG. 3.

[0209] As shown in FIG. 3, a plurality of slice sample images are included, for example Slice1 to Slice n. Each slice sample image is input into a corresponding ResNet network, and feature extraction is performed on the slice sample image through the ResNet network.

[0210] As shown in FIG. 4, each input slice sample image in the example has a size of 256×256, and is firstly subjected to a convolution kernel of 7*7 with a stride of 2 and a maximum pooling of 3*3 with a stride of 2, so that the input slice image of 256*256 is converted into a feature map with a size of 64*64, which effectively reduces the size required for storage, and then successively enters a plurality of ResNet_Block and downsample modules. Specifically, the feature map successively enters a network layer 1 composed of three ResNet_Block and a downsample module, a network layer 2 composed of three ResNet_Block and one downsample module, a network layer 3 composed of five ResNet_Block and one downsample module, and a network layer 4 composed of two ResNet_Block, and then enters an average pooling layer to finally output the sub-feature vector of each slice sample image. The sub-feature vector may be a vector of 512×1.

[0211] Next, the sub-feature vectors enter a self-attention layer, and the attention value between every two sub-feature vectors is determined by the self-attention layer. Then, the attention value between every two sub-feature vectors is input into a feature representation layer, and the sample feature corresponding to the MR sample image is obtained, based on the attention value, by the feature representation layer according to the above-mentioned formula (4) and formula (5).

[0212] As shown in FIG. 4, as shown on the right side, each ResNet_Block successively includes a 3×3 convolution with a stride of 1, Batch Norm regularization, a ReLU activation function, a 3×3 convolution with a stride of 1, and Batch Norm regularization. A structure of the downsample module is similar to ResNet_Block, and includes a 3×3 convolution with a stride of 2, Batch Norm regularization, a ReLU activation function, a 3×3 convolution with a stride of 1, Batch Norm regularization, and downsample (including a 1×1 convolution with a stride of 2 and Batch Norm regularization).

[0213] Specifically, the clinical data processing module is configured to perform feature conversion on a plurality of clinical information in the clinical sample information to obtain the sample feature corresponding to the clinical sample information.

[0214] In the example, the gender, age, histological diagnosis, tumor grading, medication information, and history of malignant tumors are used as the clinical sample information, and then mapped into the sub-feature vectors. For the sub-sample information of a character type such as gender, histological diagnosis, tumor grading, medication information, and history of malignant tumors, vg, vh, vgr, vd, and vhom may be used to represent the sub-feature vector. Specifically, vector mapping of the sub-sample information of the character type may be completed using a table lookup method.

[0215] Finally, the sub-feature vectors are fused using formula (7) to obtain the sample feature corresponding to the clinical sample information.

[0216] However, for the numerical sub-sample information such as age, the sub-feature vector corresponding to the numerical sub-sample information may be obtained according to formula (9), formula (10), and formula (11).

[0217] Specifically, the gene data processing module is configured to perform feature conversion on information of a plurality of genes in the gene sample information to obtain the sample feature corresponding to the gene sample information.

[0218] Similar to the sub-sample information of the character in the clinical sample information, vector mapping of the information of each gene may be completed using the table lookup method to obtain the sub-feature vector corresponding to the information of each gene. Then, the sub-feature vectors are fused using the formula (7) to obtain the sample feature corresponding to the gene sample information.

[0219] At S3, the sample features corresponding to the sample information of the multiple modalities are input into the fusion module. The fusion module may fuse the sample features using the formulas (12) to (14), and obtain the consistency expression value corresponding to the sample group currently input into the preset model based on the formulas (16) to (18).

[0220] At S4, the fused sample feature output by the fusion module is input into the prediction module, and the prediction module determines the predicted prognosis evaluation value based on the fused sample feature.

[0221] At S5, the loss function shown in formula (15) is constructed based on the consistency expression value output by the fusion module, the predicted prognosis evaluation value output by the prediction module, and the prognosis evaluation label corresponding to the sample group currently input into the preset model, and the parameters of the preset model are updated with the target of minimizing the difference and maximizing the consistency expression value.

[0222] When the parameters of the preset model are updated, the above-mentioned first parameter matrix, second parameter matrix, third parameter matrix, and fourth parameter matrix as well as the first parameter vector, second parameter vector, and three third parameter vectors in the parameter set may be updated synchronously, so that the accuracy of the data processing modules of the three modalities, the fusion module, and the prediction module may be affected simultaneously in one training based on two optimization targets.

[0223] At S6, after the preset model is updated for multiple times, or in the case where the difference between the predicted prognosis evaluation value and the prognosis evaluation label is less than a preset difference and the consistency expression value is greater than or equal to a preset expression value, the training is stopped. The preset model when the training is stopped is taken as the target model, through which the patient's prognosis evaluation value may be predicted.

[0224] Accordingly, referring to FIG. 5, a flowchart of steps of a method for determining a prognosis evaluation value of the present disclosure is shown. As shown in FIG. 5, the following steps may be specifically included.

[0225] At S501, information of multiple modalities of an object to be measured is acquired, the information of the multiple modalities includes an MR image, clinical information, and gene information.

[0226] At S502, the information of the multiple modalities is input into the target model to obtain a prognosis evaluation value of the object to be measured, the target model being obtained by the method for acquiring a target model described in the above-mentioned embodiment.

[0227] In the embodiment, after the target model is obtained, the patient's prognosis evaluation value may be predicted using the target model. Then, in practice, the information of the multiple modalities of the object to be measured may be acquired, and the information of the multiple modalities of the object to be measured may be consistent with the modality used for training the preset model. For example, the sample group used for training the preset model includes the MR sample image, the clinical sample information, and the gene sample information, and then the information of the multiple modalities of the object to be measured may also include the MR image, the clinical information, and the gene information of the object to be measured.

[0228] In the training process of the target model, the consistency expression value is used as a basis for updating the parameters of the preset model. The consistency expression value is used for characterizing the consistency degree of the sample features corresponding to the same target disease. Thus, with the advancement of training, the preset model may extract the prognostic factors (sample features) related to the target disease, and gradually discard the prognostic factors unrelated to the target disease in the information of each modality. Therefore, the prognostic factors selected by the model are clinically important, which may help to improve the interpretability of the model so that the result output by the target model has a high prognostic reference value.

[0229] Of course, in some alternative examples, the target model includes the data processing modules corresponding to the sample information of various modalities and the fusion module connected to the plurality of data processing modules, and the data processing module and the fusion module extract the sample feature strongly related to the expression of the target disease from the sample information of the multiple modalities. Therefore, in an application, after the target model is obtained, the data processing modules and the fusion module in the target model may be extracted separately as a feature extraction model. The feature extraction model may be configured to extract features strongly related to the target disease from the information of the multiple modalities and thus may be independently applied to the screening of the prognostic factors in the prognostic process.

[0230] The technical solutions of the embodiments of the present disclosure have the following advantages.

[0231] Firstly, through the consistency expression value, the information of different modalities may be mapped to the space reflecting important expression information of the target disease, so that the features extracted from the information of different modalities are closer in the space. Therefore, the complementarity of the information of different modalities may be improved, the influence of noise (non-important information) may be reduced, and the clinical importance thereof to the target disease is reduced, so that the target model has medical interpretability, and the predicted prognosis evaluation value has a higher medical reference value.

[0232] Secondly, the information of different modalities may be complementary, which enriches the data source of the prognostic factors and interprets the expression of the target disease from a plurality of dimensions, thus further improving the medical reference of the prognosis evaluation value.

[0233] Thirdly, the information within the same modality is combined through a self-attention mechanism, thereby enhancing the non-linear representation capability of the target model.

[0234] Fourthly, there is no need for the user to manually screen the prognostic factors multiple times, which improves the efficiency of determining the prognostic factors and reduces the manpower cost.

[0235] Based on the same inventive concept, the present disclosure also provides an apparatus for acquiring a target model. Referring to FIG. 6, a schematic structural diagram of the apparatus for acquiring a target model is shown. As shown in FIG. 6, the apparatus may specifically include the following modules:

• • [0236] a sample acquisition module 601, configured to acquire sample groups corresponding to a plurality of sample users, each of the sample group includes sample information of multiple modalities, and the sample information of the multiple modalities includes at least two of the MR sample image, the clinical sample information, and the gene sample information; and
  • [0237] a training module 602, configured to perform iterative trainings on the preset model based on the plurality of sample groups to obtain a target model, the target model being configured to predict the prognosis evaluation value of the target object, where the following steps are performed in each iterative training.

[0238] Feature extraction is performed on the sample information of the multiple modalities in the current sample group using the preset model, and a predicted prognosis evaluation value and a consistency expression value are determined based on the extracted sample features. The consistency expression value is used for characterizing the consistency degree of the sample features corresponding to the same target disease.

[0239] The parameters of the preset model are updated based on the predicted prognosis evaluation value, the prognosis evaluation label corresponding to the current sample group, and the consistency expression value.

[0240] Optionally, the training module 602 includes a parameter updating unit, and the parameter updating unit includes:

• • [0241] a difference determination subunit, configured to acquire the difference between the predicted prognosis evaluation value and the prognosis evaluation label; and
  • [0242] a parameter updating subunit, configured to update the parameters of the preset model with the target of minimizing the difference and maximizing the consistency expression value.

[0243] Optionally, the parameter updating subunit is specifically configured to:

• • [0244] construct the following loss function based on the difference and the consistency expression value:

[0245] $\mathrm{loss}={\sum }_i{\left(y_i^{\prime }-y_i\right)}^2-\mathrm{consistency};$

• • [0246] update the parameters of the preset model based on the loss value of the loss function with the target of minimizing the difference and maximizing the consistency expression value.

[0247] The loss represents the loss value, y′_i represents the predicted prognosis evaluation value, y_i represents the prognosis evaluation label, and consistency represents the consistency expression value.

[0248] Optionally, the step of determining the consistency expression value based on the extracted sample features includes the following steps.

[0249] Each of the sample features is transposed to obtain a transposed feature corresponding to each of the sample features.

[0250] For two different sample features, one of the two sample features is fused with the transposed feature corresponding to the other sample feature to obtain the corresponding fused feature value.

[0251] The consistency expression value is determined based on each fused feature value.

[0252] Optionally, the sample information of each modality includes the plurality of sub-sample information, and the performing feature extraction on the sample information of the multiple modalities in the current sample group using the preset model includes the following steps.

[0253] For the sample information of each modality, feature extraction is performed on the plurality of sub-sample information in the sample information of the modality to obtain the corresponding plurality of sub-feature vectors.

[0254] Feature fusion is performed on the plurality of sub-feature vectors corresponding to the sample information of each modality to obtain the sample feature corresponding to the sample information of the modality.

[0255] Optionally, the step of performing feature fusion on the plurality of sub-feature vectors corresponding to the sample information of each modality to obtain the sample feature corresponding to the sample information of the modality includes the following steps.

[0256] The attention value between every two sub-feature vectors is determined, the attention value being used for characterizing the degree of closeness between the two sub-feature vectors.

[0257] The plurality of sub-feature vectors are fused based on the attention value to obtain the sample feature corresponding to the sample information of the modality.

[0258] Optionally, the step of fusing the plurality of sub-feature vectors based on the attention value to obtain the sample feature corresponding to the sample information of the modality includes the following steps.

[0259] For each sub-feature vector, all the other sub-feature vectors are fused into the sub-feature vector based on attention values between the sub-feature vector and all the other sub-feature vectors to obtain the fused sub-vector for the sub-feature vector.

[0260] Re-fusion is performed on the plurality of fused sub-vectors to obtain the sample feature corresponding to the sample information of the modality.

[0261] Optionally, the parameters of the preset model include the first parameter matrix, the MR sample image includes the plurality of slice sample images, and the step of determining the attention value between every two sub-feature vectors includes the following steps.

[0262] For each slice sample image included in the MR sample image, the attention value between the sub-feature vectors corresponding to every two slice sample images is determined based on the current value of the first parameter matrix.

[0263] Optionally, the parameters of the preset model include the second parameter matrix and the third parameter matrix, and the step of performing feature fusion on the plurality of sub-feature vectors corresponding to the sample information of each modality to obtain the sample feature corresponding to the sample information of the modality includes the following steps.

[0264] For each piece of clinical sub-sample information in the clinical sample information, feature fusion is performed on the sub-feature vectors corresponding to the clinical sub-sample information based on the current value of the second parameter matrix to obtain the sample feature corresponding to the clinical sample information.

[0265] For each piece of gene sub-sample information in the gene sample information, feature fusion is performed on the sub-feature vectors corresponding to the gene sub-sample information based on the current value of the third parameter matrix to obtain the sample feature corresponding to the gene sample information.

[0266] Optionally, the parameters of the preset model include the parameter set corresponding to the clinical sample information, the clinical sample information includes the numerical sub-sample information and the non-numerical sub-sample information, and the step of performing feature extraction on the sample information of the multiple modalities in the current sample group using the preset model includes the following steps.

[0267] For the clinical sample information in the current sample group, the non-numerical sub-sample information is converted into the first sub-feature vector.

[0268] The numerical sub-sample information is mapped to the target space based on the current value of each parameter in the parameter set to obtain the corresponding second sub-feature vector, the parameters in the parameter set being used for determining the dimension of the target space and the value at each spatial point.

[0269] The first sub-feature vector and the second sub-feature vector are fused to obtain the sample feature corresponding to the clinical sample information.

[0270] Optionally, the parameter set includes the first parameter vector, the second parameter vector, and the fourth parameter matrix, and the step of mapping the numerical sub-sample information to the target space based on the current value of each parameter in the parameter set to obtain the corresponding second sub-feature vector includes the following steps.

[0271] The numerical sub-sample information is mapped to the first dimension in the target space based on the current value of the first parameter vector to obtain the mapping value of the first dimension, where the first parameter vector is used for determining the value of the spatial point of the target space in the first dimension.

[0272] The second sub-feature vector is determined based on the mapping value of the first dimension, the current value of the second parameter vector, and the current value of the fourth parameter matrix, where the second parameter vector is used for determining the value of the spatial point of the target space in the second dimension, and the fourth parameter matrix is used for assigning a parameter to each spatial position in the first dimension and the second dimension.

[0273] Optionally, the parameter set further includes a plurality of third parameter vectors, and the apparatus further includes:

[0274] a modification module, configured to correct the second sub-feature vector according to the formula as follows based on the second sub-feature vector and the plurality of third parameter vectors:

[0275] $\mathrm{va}=\frac{e^{\left(s{a}^T·a_1\right)}}{{\Sigma }_{i=1}^3{e}^{\left(s{a}^T·a_i\right)}}\times a_1+\frac{e^{\left(s{a}^T·a_2\right)}}{{\Sigma }_{i=1}^3{e}^{\left(s{a}^T·a_i\right)}}\times a_2+\frac{e^{\left(s{a}^T·a_3\right)}}{{\Sigma }_{i=1}^3{e}^{\left(s{a}^T·a_i\right)}}\times a_3.$

[0276] va is the corrected second sub-feature vector, sa is the second sub-feature vector, and a₁, a₂, and a₃ are the third parameter vectors.

[0277] The step of fusing the first sub-feature vector and the second sub-feature vector to obtain the sample feature corresponding to the clinical sample information includes the following steps.

[0278] The first sub-feature vector and the corrected second sub-feature vector are fused to obtain the sample feature corresponding to the clinical sample information.

[0279] Optionally, the parameters of the preset model include the dimension parameter matrix corresponding to the sample information of each modality, and the step of determining the predicted prognosis evaluation value and the consistency expression value based on the extracted sample features includes the following steps.

[0280] Dimension transformation is performed on the sample feature corresponding to the sample information of each modality based on the sample feature and the dimension parameter matrix corresponding to the sample information of the modality to obtain the converted sample feature.

[0281] The predicted prognosis evaluation value and the consistency expression value are determined based on the converted sample feature corresponding to each piece of the sample information of the multiple modalities.

[0282] Optionally, the preset model includes the fusion module, the prediction branch, the consistency expression branch, and the data processing module corresponding to the sample information of each modality, and the step of performing feature extraction on the sample information of the multiple modalities in the current sample group using the preset model, and determining the predicted prognosis evaluation value and the consistency expression value based on the extracted sample features includes the following steps.

[0283] For the sample information of each modality in the current sample group, feature extraction is performed on corresponding sample information using the data processing module corresponding to the sample information of the modality.

[0284] Sample features output by the data processing modules are fused using the fusion module.

[0285] The predicted prognosis evaluation value corresponding to the feature output by the fusion module is determined using the prediction branch.

[0286] The consistency expression value corresponding to the sample features output by the data processing modules is determined using the consistency expression branch.

[0287] Optionally, the gene sample information includes information of at least one gene of IDH, the chromosome 1p/19q joint deletion state, the TERT gene promoter, and the MGMT promoter methylation.

[0288] The clinical sample information includes at least one clinical information of the gender, age, histological diagnosis, tumor grading, medication information, and history of malignant tumors.

[0289] Referring to FIG. 7, a schematic structural diagram of an apparatus for determining a prognosis evaluation value is shown. As shown in FIG. 7, the apparatus includes:

• • [0290] an information acquisition module 701, configured to acquire the information of the multiple modalities of the object to be measured, the information of the multiple modalities including the MR image, the clinical information, and the gene information; and
  • [0291] an input module 702, configured to input the information of the multiple modalities into the target model to obtain the prognosis evaluation value of the object to be measured, the target model being obtained according to the method for acquiring a target model.

[0292] Based on the same inventive concept, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor when executed implements the method for acquiring a target model, or when executed implements the method for determining a prognosis evaluation value.

[0293] Referring to FIG. 8, a block diagram of an electronic device 800 of an embodiment of the present disclosure is shown. As shown in FIG. 8, the electronic device 800 provided by the embodiment of the present disclosure may be configured to perform a method for acquiring a classification model or a method for determining a mutation category of the TERT gene promoter.

[0294] The electronic device 800 may include a memory 801, a processor 802, and a computer program stored on the memory and executable on the processor. The processor 802 is configured to perform the image processing method.

[0295] As shown in FIG. 8, in an embodiment, the electronic device 800 may completely include an input apparatus 803, an output apparatus 804, and a data collection apparatus 805. When performing the image processing method of the embodiment of the present disclosure, the data collection apparatus 805 may acquire the information of the multiple modalities. Then, the input apparatus 803 may acquire information of the multiple modalities of the data collection apparatus 805, and the information of the multiple modalities may be processed by the processor 802, which may specifically perform the above-mentioned method for acquiring a target model and the above-mentioned method for determining a prognosis evaluation value. The output apparatus 804 may output the target model or may output a result of the prognosis evaluation value output by the target model.

[0296] Of course, in an embodiment, the memory 801 may include a volatile memory and a non-volatile memory. The volatile memory may be understood to be a random-access memory configured to store and maintain data. The non-volatile memory refers to a computer memory in which the stored data does not disappear when the current is turned off. Of course, the computer program of the method for acquiring a target model or the method for determining a prognosis evaluation value of the present disclosure may be stored in both the volatile memory and the non-volatile memory, or in either one of the two.

[0297] Finally, it should be noted, relational terms such as “first” and “second” are used merely to distinguish an entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or sequence between these entities or operations. Furthermore, the terms “comprising”, “including” or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus including a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the process, method, article or apparatus. Without further limitations, an element defined by the phrase “comprising a . . . ” does not exclude the presence of additional identical elements in the process, method, article or apparatus including said element.

[0298] The above is a detailed introduction to a method for acquiring a target model, a method for determining a prognosis evaluation value, a device, an apparatus and a medium provided by the present disclosure. Specific examples are used here to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and its core idea of the present disclosure. Moreover, for those skilled in the art, there will be changes in the specific implementation and application scope based on the ideas of the present application. In summary, the content of this description should not be understood as a limitation of the present application.

[0299] Other embodiments of the present disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application that follow the general principles of this application and include common knowledge or customary technical means in the technical field that are not disclosed in this application. The specification and examples be considered as exemplary only, and a true scope and spirit of the application is indicated by the following claims.

[0300] It is to be understood that the present disclosure is not limited to precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

[0301] “One embodiment”, “an embodiment” or “one or more embodiments” as used herein means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the present disclosure. In addition, please note that examples of the word “in one embodiment” herein do not necessarily all refer to the same embodiment.

[0302] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the present disclosure may be practiced without these specific details. In some instances, common methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

[0303] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word “comprising/including” does not exclude the presence of elements or steps not listed in the claim. The word “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The disclosure may be implemented by means of hardware including several distinct elements and a suitably programmed computer. In a unit claim enumerating several devices, several of these devices may be embodied by the same hardware item. The words such as “first”, “second”, and “third” as used do not indicate any order. These words may be interpreted as names.

[0304] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present disclosure, but not to limit it; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be modified, or equivalent substitutions may be made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.