Devices and methods for predicting a mobile network experience

[0001] This application is a continuation of International Patent Application No. PCT/EP2021/052729, filed on Feb. 5, 2021, the disclosure of which is hereby incorporated by reference in its entirety.

[0002] The present disclosure relates generally to the field of communication networks, and particularly to the prediction of a network experience for a mobile application using a mobile service in a mobile network.

[0003] To this end, a device is disclosed that, in an inference phase, may use a trained machine learning model for estimating a network experience score. The disclosure also presents a device that, in a training phase, may generate a database comprising a set of network experience scores of a mobile application, and may further train a machine learning model for the mobile application based on the generated database.

[0004] Conventionally, a network experience is calculated from Quality of Service (QoS) parameters, which are collected or predicted from accessible applications. Moreover, the network experience that is calculated based on conventional methods may rely on measurements and statistics from applications that represent a subscriber's use, and are measured through applications, for example, with open Application Programming Interface (APIs) or through a partnership with an application development organization.

[0005] Furthermore, the network experience score and grade are then calculated based on the QoS parameters and a scoring methodology.

[0006] However, an issue of conventional devices and methods is that the network experience measurements are limited to accessible applications. Moreover, selecting applications that will be representative of a subscriber's use or limiting testing to accessible applications will not accurately reflect the network experience.

[0007] Furthermore, another issue of conventional devices and methods is that it is not practical to partner with the many and varied application development organisations to capture application events. Moreover, there is a high cost to partnering which takes time to develop and effort to manage.

[0008] In view of the above-mentioned problems and disadvantages, embodiments of the present disclosure aim to improve conventional devices and methods. An objective is to provide a device and a method that, in an inference phase, predicts a network experience directly from network Key Performance Indicator (KPI) data. Another objective is to provide a device and a method that, in a training phase, trains a machine learning model for predicting a network experience directly from network KPI data.

[0009] The devices and methods of the present disclosure should be able to determine the network experience for the many applications using mobile services in industries enabled through Fifth Generation (5G) mobile network. These applications may use proprietary encrypted protocols for communication. Therefore, is may not be possible to access events to measure the subscriber experience of using services through these applications on a mobile network.

[0010] The objective is achieved by the embodiments of the disclosure as described in the enclosed independent claims. Advantageous implementations of the embodiments of the disclosure are further defined in the dependent claims.

[0011] In particular, the devices and methods according to embodiments of the present disclosure are configured to use a machine learning model to determine a network experience score and a grade for mobile network applications with proprietary encrypted protocols where the application events are not accessible.

[0012] A first aspect of the present disclosure provides a device for predicting a mobile network experience, the device being configured to, in an inference phase, obtain network KPI data for the mobile network by executing a set of test procedures using one or more mobile applications on one or more mobile devices in the mobile network, obtain a trained machine learning model and feed it with the network KPI data, and estimate, using the trained machine learning model, a network experience score for at least one mobile application from the one or more mobile applications based on the network KPI data.

[0013] The device may be, or may be incorporated within, an electronic device such as a personal computer, a server computer, a client computer, a laptop and a notebook computer, a tablet device, etc.

[0014] For example, during the inference phase, the device may obtain the KPI data and the trained machine learning model. Moreover, the device may estimate, using the trained machine learning model, the network experience score for the at least one mobile application based on the network KPI data. The device may obtain the network KPI data during the execution of the test procedure, for example, the network KPI data may be obtained from the mobile network itself, whilst using an application, and not from the application.

[0015] The device of the present disclosure may reduce the time and cost of drive testing. For example, the device of the first aspect may estimate the network experience score directly from the KPI data. Hence, a cost and time related to predicting the QoS parameter output used by conventional devices for predicting the network experience score may be reduced or eliminated.

[0016] Furthermore, the device of the first aspect may estimate the network experience score for applications that may be encrypted, as their events may not be accessible by the device.

[0017] The device may comprise +circuitry. The circuitry may comprise hardware and software. The hardware may comprise analog or digital circuitry, or both analog and digital circuitry. In some embodiments, the circuitry comprises one or more processors and a non-volatile memory connected to the one or more processors. The non-volatile memory may carry executable program code which, when executed by the one or more processors, causes the device to perform the operations or methods described herein.

[0018] In an implementation of the first aspect, the device is further configured to estimate, using the trained machine learning model, a network experience grade for the at least one mobile application from the one or more mobile applications based on the network KPI data.

[0019] For example, the device of the first aspect may estimate the network experience grade directly from the KPI data.

[0020] In a further implementation of the first aspect, obtaining the trained machine learning model comprises obtaining a first trained machine learning model and a second trained machine learning model for each mobile service of the one or more mobile applications; and wherein the device is further configured to estimate, using the first trained machine learning model, the network experience score for the at least one mobile application and/or estimate, using the second trained machine learning model, the network experience grade for the at least one mobile application.

[0021] In particular, if the mobile applications represent different mobile network services, then the device may obtain a machine learning model per each mobile service. For instance, the device may obtain two types of machine learning models, one type (regression) for estimating network experience scores and one type (classification) for estimating network experience grades, but there may be multiple models, one per service depending on the scoring methodology.

[0022] In a further implementation of the first aspect, the first trained machine learning model is based on a random forest machine learning model, in particular a regression model and/or the second trained machine learning model is based on a random forest machine learning model, in particular a classification model.

[0023] In a further implementation of the first aspect, the at least one mobile application is based on an encrypted protocol, providing an event that is not accessible to the device.

[0024] A second aspect of the disclosure provides a device for predicting a mobile network experience, the device being configured to, in a training phase, obtain training data by running a set of test procedures using one or more mobile applications on one or more mobile devices in the mobile network, calculate a set of network experience scores for at least one mobile application from the one or more mobile applications, based on the obtained training data, obtain network KPI data for the mobile network based on the set of test procedures, and generate a database comprising the set of network experience scores of the at least one mobile application, each network experience score being associated with a part of the network KPI data of the mobile network.

[0025] The device may be, or may be incorporated within, an electronic device such as a personal computer, a server computer, a client computer, a laptop and a notebook computer, a tablet device, etc.

[0026] For example, during the training phase, the device may obtain training data by running the set of test procedures using mobile applications that are not based on an encrypted protocol, i.e., the application events are accessible to the device. Moreover, the device may calculate the network experience scores from the QoS parameter data collected from the application in the training phase. Furthermore, the network experience scores may be collected and stored in the database, in the training phase to train a machine learning model.

[0027] In an implementation of the second aspect, the device is further configured to calculate a set of network experience grades for the at least one mobile application, based on the obtained training data, and wherein the database further comprises the set of network experience grades of the at least one mobile application, each network experience grade being associated with a part of network KPI data of the mobile network.

[0028] In a further implementation of the second aspect, the device is further configured to train a first machine learning model and a second machine learning model for the at least one mobile application based on the generated database.

[0029] In a further implementation of the second aspect, the first machine learning model is based on a random forest machine learning model, in particular a regression model, and the device is further configured to train the regression model based on feeding it with the calculated set of network experience scores, each network experience score being calculated based on a QoS parameter of the training data and associated with the part of the network KPI data of the mobile network, according to the database.

[0030] In a further implementation of the second aspect, the second machine learning model is based on a random forest machine learning model, in particular a classification model, and the device is further configured to train the classification model based on feeding it with the calculated set of network experience grades, each network experience grade being calculated based on the calculated network experience score and associated with the part of the network KPI data of the mobile network, according to the database.

[0031] In a further implementation of the second aspect, the one or more applications comprise a mobile service and are based on accessible applications providing an accessible event to the device.

[0032] In a further implementation of the second aspect, the set of test procedures comprises a user interaction comprising one or more of:

• • [0033] a web browsing procedure,
  • [0034] a video playing procedure,
  • [0035] a social media uploading procedure,
  • [0036] a social media downloading procedure,
  • [0037] a virtual reality procedure,
  • [0038] an augmented reality procedure,
  • [0039] a cloud game procedure, and
  • [0040] an entertainment procedure.

[0041] In a further implementation of the second aspect, the training data for the mobile service comprises one or more of:

• • [0042] a QoS parameter for an accessibility of the at least one mobile application,
  • [0043] a QoS parameter for a retain-ability of the at least one mobile application,
  • [0044] a QoS parameter for a quality of the at least one mobile application,
  • [0045] a minimum value of an allowed network experience score, and
  • [0046] a maximum value of an allowed network experience score.

[0047] In a further implementation of the second aspect, the network KPI data comprises one or more of:

• • [0048] latency data,
  • [0049] packet loss data,
  • [0050] delay variance data,
  • [0051] jitter data,
  • [0052] a data rate,
  • [0053] bandwidth data,
  • [0054] reference signal receive power data,
  • [0055] a signal-to-interference-plus-noise ratio data,
  • [0056] a round trip time.

[0057] In a further implementation of the second aspect, each network experience score has a non-negative integer value.

[0058] In a further implementation form of the second aspect, each network experience grade is one of:

• • [0059] an outstanding grade,
  • [0060] a very good grade,
  • [0061] a good grade,
  • [0062] a satisfactory grade, and
  • [0063] a sufficient grade.

[0064] A third aspect of the disclosure provides a method for predicting a mobile network experience, the method comprising, in an inference phase, obtaining network KPI data for the mobile network by executing a set of test procedures using one or more mobile applications on one or more mobile devices in the mobile network, obtaining a trained machine learning model and feeding it with the network KPI data, and estimating, by the trained machine learning model, a network experience score for at least one mobile application from the one or more mobile applications based on the network KPI data.

[0065] In an implementation of the third aspect, the method further comprises estimating, using the trained machine learning model, a network experience grade for the at least one mobile application from the one or more mobile applications based on the network KPI data.

[0066] In a further implementation of the third aspect, obtaining the trained machine learning model comprises obtaining a first trained machine learning model and a second trained machine learning model for each mobile service of the one or more mobile applications; and wherein the method further comprises estimating, using the first trained machine learning model, the network experience score for the at least one mobile application and/or estimate, using the second trained machine learning model, the network experience grade for the at least one mobile application.

[0067] In a further implementation of the third aspect, the first trained machine learning model is based on a random forest machine learning model, in particular a regression model and/or the second trained machine learning model is based on a random forest machine learning model, in particular a classification model.

[0068] In a further implementation of the third aspect, the at least one mobile application is based on an encrypted protocol, providing an event that is not accessible to the device.

[0069] The method of the third aspect achieves the advantages and effects described for the device of the first aspect.

[0070] A fourth aspect of the disclosure provides a method for predicting a mobile network experience, the method comprising, in a training phase, obtaining training data by running a set of test procedures using one or more mobile applications on one or more mobile devices in the mobile network, calculating a set of network experience scores for at least one mobile application from the one or more mobile applications, based on the obtained training data, obtaining network KPI data for the mobile network based on the set of test procedures, and generating a database comprising the set of network experience scores of the at least one mobile application, each network experience score being associated with a part of the network KPI data of the mobile network.

[0071] In an implementation of the fourth aspect, the method further comprises calculating a set of network experience grades for the at least one mobile application, based on the obtained training data, and wherein the database further comprises the set of network experience grades of the at least one mobile application, each network experience grade being associated with a part of the network KPI data of the mobile network.

[0072] In a further implementation of the fourth aspect, the method further comprises training a first machine learning model and a second machine learning model for the at least one mobile application based on the generated database.

[0073] In a further implementation form of the fourth aspect, the first machine learning model is based on a random forest machine learning model, in particular a regression model, and the method further comprises training the regression model based on feeding it with the calculated set of network experience scores, each network experience score being calculated based on a QoS parameter of the training data and associated with the part of the network KPI data of the mobile network, according to the database.

[0074] In a further implementation of the fourth aspect, the second machine learning model is based on a random forest machine learning model, in particular a classification model, and the method further comprises training the classification model based on feeding it with the calculated set of network experience grades, each network experience grade being calculated based on the calculated network experience score and associated with the part of the network KPI data of the mobile network, according to the database.

[0075] In a further implementation of the fourth aspect, the one or more applications comprise a mobile service and are based on accessible applications providing an accessible event to the device.

[0076] In a further implementation of the fourth aspect, the set of test procedures comprises a user interaction comprising one or more of:

• • [0077] a web browsing procedure,
  • [0078] a video playing procedure,
  • [0079] a social media uploading procedure,
  • [0080] a social media downloading procedure,
  • [0081] a virtual reality procedure,
  • [0082] an augmented reality procedure,
  • [0083] a cloud game procedure, and
  • [0084] an entertainment procedure.

[0085] In a further implementation of the fourth aspect, the training data for the mobile service comprises one or more of:

• • [0086] a QoS parameter for an accessibility of the at least one mobile application,
  • [0087] a QoS parameter for a retain-ability of the at least one mobile application,
  • [0088] a QoS parameter for a quality of the at least one mobile application,
  • [0089] a minimum value of an allowed network experience score, and
  • [0090] a maximum value of an allowed network experience score.

[0091] In a further implementation of the fourth aspect, the network KPI data comprises one or more of:

• • [0092] latency data,
  • [0093] packet loss data,
  • [0094] delay variance data,
  • [0095] jitter data,
  • [0096] a data rate,
  • [0097] bandwidth data,
  • [0098] reference signal receive power data,
  • [0099] a signal-to-interference-plus-noise ratio data, and
  • [0100] a round trip time.

[0101] In a further implementation of the fourth aspect, each network experience score has a non-negative integer value.

[0102] In a further implementation form of the fourth aspect, each network experience grade is one of:

• • [0103] an outstanding grade,
  • [0104] a very good grade,
  • [0105] a good grade,
  • [0106] a satisfactory grade, and
  • [0107] a sufficient grade.

[0108] The method of the fourth aspect achieves the advantages and effects described for the device of the second aspect.

[0109] A fifth aspect of the present disclosure provides a computer program comprising a program code for performing the method according to the third aspect or the fourth aspect or any of their implementations.

[0110] A sixth aspect of the present disclosure provides a non-transitory storage medium storing executable program code which, when executed by a processor, causes the method according to the third aspect or the fourth aspect any of their implementation forms to be performed.

[0111] It has to be noted that the devices, elements, units and means described in the present application could be implemented in software or hardware elements or any kind of combination thereof. The steps which are performed by the various entities described in the present application, as well as the functionalities described to be performed by the various entities, are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof.

[0120] FIG. 1 depicts a schematic view of a device 100 for predicting a mobile network experience, according to an embodiment of the disclosure.

[0121] The device 100 may be, for example, an electronic device such as a computer.

[0122] The device 100 is configured to, in an inference phase, obtain network Key Performance Indicator (KPI) data 101 for the mobile network 1 by executing a set of test procedures using one or more mobile applications on one or more mobile devices in the mobile network 1.

[0123] The device 100 is further configured to, in the inference phase, obtain a trained machine learning model 102 and feed it with the network KPI data 101.

[0124] The device 100 is further configured to, in the inference phase, estimate, using the trained machine learning model 102, a network experience score 103 for at least one mobile application from the one or more mobile applications based on the network KPI data 101.

[0125] The device 100 may comprise processing circuitry (not shown) configured to perform, conduct, or initiate the various operations of the device 100 described herein. The processing circuitry may comprise hardware and software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the device 100 to perform, conduct or initiate the operations or methods described herein.

[0126] FIG. 2 depicts a schematic view of a device 200 for predicting a mobile network experience, according to an embodiment of the disclosure.

[0127] The device 200 may be, for example, an electronic device such as a computer.

[0128] The device 200 is configured to, in a training phase, obtain training data 201 by running a set of test procedures using one or more mobile applications on one or more mobile devices in the mobile network 1.

[0129] The device 200 is further configured to, in the training phase, calculate a set of network experience scores 203 for at least one mobile application from the one or more mobile applications, based on the obtained training data 201.

[0130] The device 200 is further configured to, in the training phase, obtain network KPI data 202 for the mobile network 1 based on the set of test procedures.

[0131] The device 200 is further configured to, in the training phase, generate a database 204 comprising the set of network experience scores 203 of the at least one mobile application, each network experience score 203 being associated with a part of network KPI data 202 of the mobile network 1.

[0132] The device 200 may comprise processing circuitry (not shown) configured to perform, conduct, or initiate the various operations of the device 100 described herein. The processing circuitry may comprise hardware and software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the device 200 to perform, conduct, or initiate the operations or methods described herein.

[0133] Reference is now made to FIG. 3 which is a schematic view of the mobile network 1 comprising the device 100 and the device 200, according to an embodiment of the disclosure.

[0134] In particular, the device 200 may build a supervised machine learning model per service based on training data 201 from one accessible application, e.g., the device 200 may train the machine learning model to associate the network score 203 with a pattern of network KPIs 202.

[0135] Furthermore, the device 100 may use the machine learning model 102 to predict the network experience score 103 or the network experience grade of a subscriber for any application using that mobile service.

[0136] For example, the device 100 may predict the network experience score 103 and the network experience grade for any application. Furthermore, the grade e.g., ranging from satisfactory to outstanding, is universally clear. The network experience score 103 e.g., a number is more useful for mobile network testers, optimization engineers etc.

[0137] The device 100 and/or the device 200 may enable predicting a network experience. Moreover, the prediction of a network experience score and grade for mobile applications may be performed, without the need to access the application measurements or events. Prediction may be based on a machine learning model that has learned the pattern of network KPI data 202 for a particular network experience score and grade. Once the machine learning model is created, it may determine, based on the KPI data alone, a predicted network experience for that application on a particular mobile network.

[0138] The network experience score and grade may be a value from 0 to x and may be suitable for drive testing and optimization engineers (Note: x is dependent on the maximum score for the service in the scoring methodology).

[0139] Further, the network experience grade is one of outstanding, very good, good, satisfactory, sufficient and suitable for customer care, operations and management reports.

[0140] The device 100 and/or the device 200 may provide the following advantages:

[0141] Ability to test and compare applications without a need to access proprietary encrypted protocols or develop costly partnerships.

[0142] Removes uncertainty in selecting/deciding which is the most representative application for a service to measure a network experience.

[0143] Test many applications across many use cases and services.

[0144] Better representation of network performance as experienced by the user.

[0145] Experiment with network parameters to predict an application experience or connectivity requirements.

[0146] Reference is now made to FIG. 4 which is a schematic view of a flow chart of a procedure 400 for training a machine learning model and predicting a network experience for a mobile application, according to an embodiment of the disclosure.

[0147] The flowchart of the procedure 400 outlines the workflow in two phases:

• • [0148] Training phase (Steps 401 to 405): building the supervised machine learning model using training data.
  • [0149] Inference Phase (steps 406 to 409): applying the machine learning model to network data to predict network experience.
    Training Phase: Steps 401 to 405

[0150] At 401, the device 200 may obtain training data 201 by running a set of test procedures using one or more mobile applications on one or more mobile devices in the mobile network 1.

[0151] The device 200 may execute test procedures using an accessible application. The test procedure may include: web browsing, video playing, social media uploads, cloud gaming, VR entertainment, retail, fitness, etc.

[0152] At 402, the device 200 may collect application measurements and may further obtain KPI data 202 for the mobile network based on the set of test procedures performed at 401.

[0153] The device 200 may collect test results per service i.e., QoS parameters for accessibility retain ability, quality, etc.

[0154] Moreover, the device 200 may collect network KPI data including but not limited to: latency, packet loss, delay variance, jitter, data rate, bandwidth, rsrp, sinr, and round trip time.

[0155] At 403, the device 200 may calculate a set of network experience scores 203 for the mobile application and may further calculate a set of network experience grades for the application.

[0156] At 404, the device 200 may train a first machine learning model and a second machine learning model for the mobile application.

[0157] For example, the device 200 may train a random forest machine learning algorithm.

[0158] The prediction range may be from highest to lowest labels in the training data. The min and max score in the training data may determine the range supported by the model.

[0159] Moreover, because there are different scoring limits for different services a model per service may be required.

[0160] At 405, the device 200 may obtain a network experience prediction model, i.e., the trained machine learning model.

[0161] Additionally, there is a location or geographical context to the network experience whereby the scoring and/or grading contains location information. For instance, depending on the location type, e.g., urban, rural etc., there may be a different scoring model.

Inference Phase Steps 406 to 409

[0162] At 406, the device 100 may execute a set of test procedures using one or more mobile applications on one or more mobile devices in the mobile network 1.

[0163] For example, the device 100 executes test procedures for any application. The test procedure may include web browsing, video playing, social media uploads, cloud gaming, VR entertainment, etc.

[0164] At 407, the device 100 may obtain network KPI data 101 for the mobile network 1.

[0165] For example, the device 100 may collect network KPI including but not limited to: latency, packet loss, delay variance, jitter, data rate, bandwidth, rsrp, sinr, and round trip time.

[0166] At 408, the device 100 may apply KPI data for the appropriate scoring service model and may further predict a network experience score for the mobile application.

[0167] At 409, the device 100 may apply network KPI data for the appropriate grading service model and may further predict network experience grade for this application.

[0168] Reference is now made to FIG. 5, which is a schematic view of a diagram illustrating an exemplary component of the device 100 and/or the device 200.

[0169] In particular, FIG. 5 shows a high level solution architecture that may include one or more of:

• • [0170] Data generation component e.g., mobile device running application tests;
  • [0171] Data collection component;
  • [0172] Data processing component;
  • [0173] Data storage component;
  • [0174] Data Analytics component;
  • [0175] Machine learning model; and
  • [0176] Network experience analysis and exposure component.

[0177] FIG. 6 shows a method 600 according to an embodiment of the disclosure for predicting a mobile network experience. The method 600 may be carried out by the device 100, as it described above.

[0178] The method 600 comprises, in an inference phase, a step 601 of obtaining network Key Performance Indicator, KPI, data 101 for the mobile network 1 by executing a set of test procedures using one or more mobile applications on one or more mobile devices in the mobile network 1.

[0179] The method 600 further comprises, in the inference phase, a step 602 of obtaining a trained machine learning model 102 and feeding it with the network KPI data 101.

[0180] The method 600 further comprises, in the inference phase, a step 603 of estimating, by the trained machine learning model 102, a network experience score 103 for at least one mobile application from the one or more mobile applications based on the network KPI data 101.

[0181] FIG. 7 shows a method 700 according to an embodiment of the disclosure for predicting mobile network experience. The method 700 may be carried out by the device 200, as it described above.

[0182] The method 700 comprises, in a training phase, a step 701 of obtaining training data 201 by running a set of test procedures using one or more mobile applications on one or more mobile devices in the mobile network 1.

[0183] The method 700 further comprises, in the training phase, a step of calculating 702 a set of network experience scores 203 for at least one mobile application from the one or more mobile applications, based on the obtained training data 201.

[0184] The method 700 further comprises, in the training phase, a step 703 of obtaining network KPI data 202 for the mobile network 1 based on the set of test procedures.

[0185] The method 700 further comprises, in the training phase, a step 704 of generating a database 204 comprising the set of network experience scores of the at least one mobile application, each network experience score 203 being associated with a part of the network KPI data 202 of the mobile network 1.

[0186] The present disclosure has been described in conjunction with various embodiments as examples as well as implementations. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed disclosure, from the studies of the drawings, this disclosure and the independent claims. In the claims as well as in the description the word “comprising” does not exclude other elements or steps and the indefinite article “a” or “an” does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.