Location based services quality assessment

[0001] This application is a continuation-in-part application of U.S. patent application Ser. No. 11/227,972, filed Sep. 15, 2005, entitled “Assessing Performance and Quality of a Mobile Communication Service”, which is incorporated herein by reference in its entirety.

[0002] The technical field is generally related to communication systems and more specifically related to assessing the quality of a service provided via a communications system.

[0003] Maintaining quality of service in the eyes of customers is an important factor in services industries. This is the case in the wireless/mobile communications industry, as well as the traditional wireline telephone industry where customers often base their decisions to join, or stay with a particular service provider based on the quality of the services provided. For example, with respect to wireless telephones, events such as recurring dropped calls, poor sound quality during calls and unexpected unavailability of service may drive customers to seek a new service provider, especially given rising standards for call quality.

[0004] Currently, various techniques exist for monitoring quality of service in the wireless communications industry. Some of these techniques, such as PESQ (Perceptual Evaluation of Speech Quality) obtain objective quality of service information. For example, PESQ measures voice quality by comparing an input test signal with the signal output across a connection. Another objective technique, ITU-T E-model (e.g., ITU-T G. 107) predicts conversational MOS (mean opinion score) from IP networks and/or terminal parameters. Subjective techniques also exist where test mobile devices are established to monitor sound quality. Service providers often arrange for such testing immediately following deployment of new networks, new network elements (such as base stations) and network changes or upgrades, etc.

[0005] The techniques described above are often difficult to implement, may be limited in their capability to monitor an entire network, and may be expensive, especially in the case of call quality monitoring techniques that utilize specialized infrastructure in the wireless environment. Many of these techniques may also cause unwanted load on the network. In addition, it is often not practical to implement such techniques on a regular basis. In addition, current quality monitoring techniques are difficult to implement throughout the entire network, especially when networks may span large and diverse geographical area and the test devices have to be frequently mobile to simulate as much as the real user experience. For example, current quality monitoring techniques may be implemented so that a single sector is used to report on the quality of an entire region. Accordingly, it may be difficult to accurately monitor all areas of the network using such techniques.

[0006] Described herein is a means for user feedback concerning observed accuracy and quality of location-based services in a wireless networked system. In one embodiment, a wireless network provides a capability to accept user-provided quality feedback information concerning a target location data provided by the network in response to a query for location information from the user. Feedback data provided by the user can be correlated to other aspects of system operation such as the preset location of the wireless receiving device, the call traffic of the system, and other relevant information. Reports may be generated to assist a system engineer or system administrator in troubleshooting and adjusting the wireless network to lessen or eliminate the target location data provided by the wireless system.

[0018] In the drawings, the same reference numbers identify identical or substantially similar elements or acts. To facilitate the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced (e-g., element 204 is first introduced and discussed with respect to FIG. 2).

[0019] Location based services quality assessment will now be described with respect to various embodiments. The following description provides specific details for a thorough understanding of, and enabling description for, these embodiments. However, one skilled in the art will understand that location based services quality assessment may be practiced without these details. In other instances, well-known structures and functions have not been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0020] It is intended that the terminology used in the description presented be interpreted in its broadest reasonable manner, even though it is being used in conjunction with a detailed description of certain specific embodiments. Certain terms may even be emphasized below; however, any terminology intended to be interpreted in any restricted manner will be overtly and specifically defined as such in this Detailed Description section.

[0021] FIGS. 1-3 and the following discussion provide a brief, general description of a suitable computing/network environment in which the call quality monitoring can be implemented. Although not required, aspects of the call quality monitoring are described in the general context of computer-executable instructions, such as routines executed by a general-purpose computer, e.g., a server computer, wireless device, or personal computer. Those skilled in the relevant art will appreciate that location based services quality assessment can be practiced with other communications, data processing or computer system configurations, including Internet appliances, hand-held devices (including PDAs), wearable computers, all manner of cellular or mobile phones, multi-processor systems, microprocessor-based, ASIC-based, or PGA-based systems, programmable consumer electronics, set-top boxes, network PCs, minicomputers, mainframe computers, and the like. Indeed, the terms “computer,” “device,” and “component” are generally used broadly and interchangeably, and refer to any of the above devices and systems, as well as any data processor.

[0022] Aspects of the call quality monitoring can be embodied in a special purpose computer or data processor that is specifically programmed, configured, or constructed to perform one or more of the computer-executable instructions explained in detail herein. Aspects of the call quality monitoring can also be practiced in distributed computing environments where tasks or modules are performed by remote processing devices, which are linked through a communications network, such as a LAN, WAN, or the Internet. In a distributed computing environment, program modules may be located in both local and remote memory storage devices (e.g., including memory associated with field programs, gate arrays, EPROM memory, etc.).

[0023] Aspects of the call quality monitoring may be stored or distributed on computer-readable media, including magnetically or optically readable computer discs, hard-wired or preprogrammed in chips (e.g., EEPROM semiconductor chips), nanotechnology memory, photonic memory, biological-based memory, or other data storage media, including but not limited to any one or more of the following: magnetic, optical, electro optical, semiconductor, and super conducting. Indeed, computer implemented instructions, data structures, screen displays, and other data under aspects of location based services quality assessment may be distributed over the Internet or over other networks (including wireless networks), on a propagated signal on a propagation medium (e.g., an electromagnetic wave(s), a sound wave, etc.) over a period of time, or may be provided on any analog or digital network (packet switched, circuit switched, or other scheme). Those skilled in the relevant art will recognize that portions of location based services quality assessment can reside on a server computer, while corresponding portions reside on a client computer such as a mobile or portable device, and thus, while certain hardware platforms are described herein, aspects of location based services quality assessment are equally applicable to nodes on a network.

[0024] Referring to FIG. 1, a system 100 on which the quality of service assessment system may be implemented is shown. The system of FIG. 1 is an example of a GPRS (general packet radio service) system based on GSM (global system for mobile communication). However, location based services quality assessment may be implemented on other types of systems and technologies including a variety of cellular systems, UMTS, HPDSA, . . . , as well as other broadcast, multicast and point to point communication systems that are any combination of simplex, half duplex, full duplex including terrestrial broadcast, cable, satellite broadcast, satellite cell, and any one or any combination of PAN, LAN, WAN, MAN, or/and other network(s) including Bluetooth, and 802.11 family. The system 100 includes a mobile device 102 (e.g., mobile phone, PDA, wireless laptop, etc.) in communication with a base station 104. A base station controller 106 in communication with a serving GPRS support node (SGSN) 108 and a gateway GPRS support node (GGSN) 110 together support packet switched transactions, which are handled separately from circuit switched traffic that is supported by a mobile switching center (MSC) 111. The MSC 111 also serves as an access point for the Public Switched Telephone Network.

[0025] The SGSN 108, GGSN 110, and MSG 111 interact with a home location register 112 (HLR). In some embodiments, the HLR 112 is the primary database of permanent subscriber customer information for the service provider's mobile network. In the context of activated devices, the HLR 112 may contain pertinent user information, including address information, account status, and preferences. In some embodiments, a visiting location register (VLR) 114 manages requests from out-of-area subscribers who are out of the area covered by their home system.

[0026] In the illustrated embodiment, the system 100 includes components associated with quality of service assessment including a mobile service provider administration system 120. The mobile service provider administration system 120 may include a quality monitoring system 122, a call quality database 124, a customer care system 126, and a billing system 128, described in more detail with respect to FIG. 3. Customers may interact with the mobile service provider administration system 120 and its various components via the mobile device 102 and a wireless services network 116, as well as through other means, such as the internet 118.

[0027] FIG. 2 shows a block diagram of a typical mobile communication device 200, such as a mobile handset. While a mobile phone is shown as the mobile communication device in FIG. 1, those skilled in the relevant art will appreciate that location based services quality assessment can be practiced with other devices and configurations, including Internet appliances, hand-held devices, wearable computers, multi-processor systems, microprocessor-based or programmable consumer electronics, set-top boxes, PDAs, portable laptop computers, and the like. The term “mobile device” is intended to include all such devices.

[0028] The mobile device 200 has one or more internal or external antennas 202 for receiving and transmitting electromagnetic signals such as radio frequency signals. A transceiver 204 is connected to the antenna(s) 202 and typically provides modulation and demodulation of the transmitted and received signals, respectively. A processor unit 206 connected to the transceiver 204 may comprise a signal processor, microprocessor, ASIC, or other control and processing logic circuitry. The processor unit 206 may perform signal coding, data processing, input output processing, power control, and other functions necessary for implementing a mobile communication device. A customer may provide input to the processor unit 206 via a keypad 208, microphone 210, or display touchpad 212. In turn, the processor unit 206 may provide information to the customer via the display touchpad 212 or a speaker 214.

[0029] The processor unit 206 may access information from, and store information in, a nonremovable memory 216 or a removable memory 218. The non-removable memory 216 may consist of RAM, ROM, EPROM, a hard disk, or other memory storage technologies. The removable memory 218 may consist of Subscriber Identity Module (SIM) cards, which are well known in GSM communication systems, or other well-known memory storage technologies, such as “smart cards.” Various applications, including text message applications 220 and quality reporting applications 220, could be implemented in either the removable memory 218 or the non-removable memory 216. For example, the applications may include a user interface application (e.g., a Java applet) that allows a user to rate a call or other transaction (e.g., by pushing a particular button). The applications may allow rating in real time (e.g., while a call is in progress) or, alternatively, after call has been completed. In some embodiments, an application used to rate a call or other communication may be located on a device that is separate from the device used to place the call itself (e.g., a smart device with GPS compatibility so that the precise location of the user can be identified if the phone itself does not have such capabilities).

[0030] In some embodiments, a device location component 224 allows the location of the device to be known to the wireless service provider, so that the wireless service provider can use this information (or pass it along) for the purpose of assessing call quality factors.

[0031] Referring to FIG. 3, a more detailed view of the mobile service provider administration system 120 of FIG. 1 is shown. The mobile service provider administration system 120 may include a call quality monitoring system 122 (also shown in FIG. 1). Various components of the quality monitoring system 122 may include a quality reporting customer interface 302 and a quality reporting processing and analysis component 304. Both of these components may communicate with a call quality database 124 (also shown in FIG. 1). For example, in some embodiments the quality reporting customer interface 302 may provide a way for customers to interact with the quality monitoring system 122 and provide information that can then be processed by the quality reporting processing and analysis component 304.

[0032] Both the data collected from the customer using the quality reporting customer interface 302 and the data resulting from processing by the quality reporting processing and analysis component 304 may be stored in the call quality database 124. In some embodiments the quality reporting customer interface 302 may be limited to a physical device interface (as opposed to possessing application-based user interface characteristics). For example, if the user's mobile device includes a comprehensive interface for providing quality reporting information, the quality reporting customer interface 302 may simply be an interface to retrieve information from the device. In contrast, if the user's mobile device does not include a comprehensive reporting interface, the quality reporting customer interface 302 may include various features that allow users to answer questions and/or provide statements about the quality of their calls. For example, in some embodiments, quality reporting customer interface 302 may include automated telephone questioning systems, a web server and related applications for providing online questionnaires, etc.

[0033] In some embodiments, the quality monitoring system 122 may also include other quality control components 306. For example these other quality control components may use objective standards for providing quality control information. Examples include PESQ systems, systems employing test mobile devices, etc. In some embodiments, the information collected by the other quality control components 306 may be compared with information obtained and retrieved by the quality reporting customer interface 302 and processed by the quality reporting processing and analysis component 304.

[0034] As described with respect to FIG. 1, the mobile service provider administration system 120 may also include a customer care system 126 and a billing system 128, which may optionally interact with the quality monitoring system 122 to address customer concerns such as dropped calls, credits to service plans, etc.

[0035] Unless described otherwise below, aspects of location based services quality assessment may be practiced with conventional systems. Thus, the construction and operation of the various blocks shown in FIGS. 1-3 may be of conventional design, and need not be described in further detail herein to make and use location based services quality assessment, because such blocks will be understood by those skilled in the relevant art. One skilled in the relevant art can readily make any modifications necessary to the blocks in FIGS. 1-3 (or other embodiments or Figures) based on the detailed description provided herein.

[0036] Referring to FIG. 4A, a high level call quality assessment routine 400 may provide a means for customers to report on call quality in accordance with some embodiments. At block 401a call is initiated, which may include initiation at the network level and/or at the device level. At block 402 the call is in progress. At block 403 the call is terminated. For example, the user may have hung up the device to end the call, or the call may have been dropped due to inadequate service.

[0037] At block 404 a quality reporting option is initialized upon termination of the call. For example, the user may receive a message requesting input on the quality of the call or may be given the option to make a call to an automated quality monitoring reporting system. Many options for reporting are available without departing from the scope of location based services quality assessment. At block 405 the quality monitoring system collects call information. At block 406 the quality monitoring system performs exploratory and/or automated data analysis. As a result of this data analysis, the quality monitoring system may identify issues not detected by objective assessment of individual network modes (block 406a); quantify actual customer experience across the entire communication network (block 406b); assess the impact of changes made to the communication network (block 406c); identify, localize, and resolve call quality degradation issues (block 406d), etc.

[0038] FIG. 4B shows a high level call quality assessment routine 420, similar to the routine 400 of FIG. 4A, but where the call quality information is collected from the user while the call is still in progress. For example at block 421a call is initiated, which may include initiation at the network level and/or at the device level. At block 422 the call is in progress. At block 423, the quality monitoring system collects call information during the call. For example, the user may be able to push a button on the phone each time he or she feels that call quality during the call is below an acceptable level. At block 424 the call is terminated. For example, the user may have hung up the device to end the call, or the call may have been dropped due to inadequate service. At block 425 the quality monitoring system performs exploratory and/or automated data analysis based on the information collected at block 423. Some of this analysis may also occur while the call is in progress. As a result of this data analysis, the quality monitoring system may identify issues not detected by objective assessment of individual network modes, quantify actual customer experience across the entire communication network, assess the impact of changes made to the communication network, identify, localize, and resolve call quality degradation issues, etc.

[0039] FIG. 5 is a flow diagram showing an example of a routine 500 for collecting quality information is collected by an automatic callback feature. At block 501 the call is terminated. At block 502 the quality monitoring system sends a message to the device to perform auto callback upon completion of a call. Alternatively, the device may be programmed to automatically place such a call, depending on the conditions present at the device (e-g., termination of a call by the user, termination of a call due to loss of service or other network problem, etc.

[0040] At block 503 the quality monitoring system receives a callback from the mobile device. When this occurs, mobile device may ring as the call is being placed automatically by the mobile device. In some cases the user may have the option of hanging up or discontinuing the feedback call if the user does not want to provide input at this time. At block 504 the quality monitoring system receives user input via the automatic call. The routine 500 ends after block 504.

[0041] FIG. 6 is a flow diagram showing an example of a callback option routine 600 for providing an opportunity for the user to optionally provide feedback on a call that was recently made. In this routine 600, the user has the option of whether or not to continue with a reporting call before the call is made. At block 601a call is terminated. At block 602 the quality monitoring system sends a message to the device to prompt the user for a callback. Alternatively, the device may be programmed to display such a message without a prompt from the quality monitoring system. For example the user may receive a message displayed on his or her device screen requesting whether he or she would like to call back and report on call quality at the current time.

[0042] At decision block 603, if the user selects to go forward with the callback option, the routine 600 continues at block 604. Otherwise the routine ends without reporting. At block 604 the routine receives an auto callback call from the device in response to the user's selection to provide feedback. At block 605 the quality monitoring system receives user input. After block 605 the routine ends.

[0043] FIG. 7 is a flow diagram showing an example of a routine 700 for a post-call questionnaire, where the questionnaire is executed by a program on the mobile device itself. At block 701 the call is terminated. At block 702 the routine 700 begins executing at the mobile device. At block 703 the routine 700 poses a next question to the user. At decision block 704 if the routine 700 has posed a last question to the user, the routine 700 continues at block 705. Otherwise, the routine 700 loops back to block 703 to pose the next question to the user. At block 705, assuming all questions have been posed to the user, the routine 700 sends a response to the call quality monitoring system. For example, the routine 700 may provide data comprising a completed series of questions or statements by the user on call quality.

[0044] FIG. 8 is a flow diagram showing an example of a routine 800 for presenting a post call questionnaire to the user of a mobile device, where the routine 800 is executed from the quality monitoring system. At block 801 the call is terminated. At block 802 the routine 800 contacts the device to initiate connection with the questionnaire application. At block 803 the routine 800 provides a next question to the device. At block 804 the routine 800 receives a response from the device (e.g., provided by the user as a response to a question in the questionnaire). At decision block 805, if the last question has been posed, the routine 800 ends. Otherwise, the routine 800 moves back to block 803 to provide the next question to the device.

[0045] FIG. 9 is a display diagram showing various examples of providing a user of a mobile device with an option to answer questions relating to call quality. With respect to display 902, a call has been dropped due to a service problem, and the user is asked:

[0046] “Would you like to report on call quality now?” In this example, the user may select either “yes” or “no” using appropriate device keys.

[0047] In another example shown in display 904, a message provides: “Rate the quality of your last call now and receive free airtime minutes.” The user may then respond “yes” or “no” as to whether he or she wishes to rate the call at the present time.

[0048] In a third example shown in display 906, a message provides three rating options: “Select 1 to report on call quality now”; “Select 2 to report on call quality later”; and “Select 3 to opt out of reporting call quality in relation to the current call.” If the user selects the first option, the device may present a text questionnaire to the user or make a call to an automated quality reporting system, as described in more detail in the preceding flow diagrams and with respect to FIG. 10. Alternatively if the user selects to report on call quality later, the user may be given the option to provide feedback at a later time via any one of a number of means (e.g., Internet questionnaire, automated, call, text message on device screen, etc.).

[0049] FIG. 10 is a display diagram showing an example of a user interface for reporting call quality. Display 1002 shows a text mode where the user is asked to report on call quality after a dropped call using text messaging or similar means. For example the user may be requested to answer the question: “Did the signal strength vary during your call?” To this the user may respond either “yes” or “no” using appropriate buttons on his or her device.

[0050] Display 1004 illustrates reporting on call quality using an audio mode (e-g., automated telephone system). For example, the user may be asked to “describe the quality of your last call.” The user's response may then be recorded and or automatically processed. In another example, the user may be asked: “What was the signal strength just prior to the time that the call was dropped? For poor select or say 1, or moderate . . . ” Another question that the user may be asked is “Have you had problems with dropped calls in this area before?”

[0051] While specific examples are given here for the purpose of illustration, regardless of the mode used (e.g., text mode 1002, in an audio mode 1004, or in another mode not illustrated here), any number of questions may be asked to the user for quality reporting. While by no means all inclusive, the table below shows examples of other types of questions that may be posed to users in determining call quality:

[0052]

Sound Quality
On a scale of 1-5, with 5 being the best quality sound, how would you rate the quality of the
sound of your call?
Were you able to hear the person on the other end of the line clearly during the call? Never?
Rarely? Most of the time? Always?
Was the person on the other end of the line able to hear you during the call? Never? Rarely?
Most of the time? Always?
Do you have repeated problems with sound quality when placing calls in this area?
Noise
On a scale of 0-5 with 0 being no detectable background noise, how would your rate the amount
of background noise of your call?
How often were you able to hear unwanted background noise during your call? Never? Rarely?
Most of the time? Always?
What was the severity of background noise during your call? Greatly affected ability to hear
content of call. Somewhat affected ability to hear content of call. Did not affect the ability to
hear content of call but was annoying. Was audible but not distracting. No detectable background
noise present
Do you have repeated problems with sound quality when placing calls in this area?
Dropped Call
Did the signal strength vary during your call?
What was the signal strength at the time the call was placed?
What was the signal strength just prior to the time the call was dropped?
Have you had problems with dropped calls in this area before?
How many calls do you make in the area that the call was dropped?
How many times did you attempt to reconnect the same call?
Subjective Reponses
Describe the quality of your last call.
Describe your overall satisfaction with the quality of your last call.
Describe how the quality of your last call could have been improved.
Describe your overall experience with your wireless communication service provider.
Describe your overall satisfaction wit your wireless communication service provider.
Other
Did you have dial tone problems when making this call? Press 1 for yes. Press 2 for no.
Rate the call set up time for this call.
Please indicate if there were issues disconnecting from this call.
Please indicate if there were problems handing off between cells during this call.

[0053] In addition to the above, other factors and techniques may be used in rating calls. For example, various different types of rating scales may be implemented. In another example, users may be provided with descriptive icons or graphics to select from to make call rating fast and easy.

[0054] In another aspect of location based services quality assessment, location service quality may be assessed from a user perspective. In this aspect, the user can provide feedback on the perceived quality of location services provided by a wireless telephone network that are received on a user wireless device. It is possible that the present location of a user, as provided by her mobile device, is incorrect. And thus, location services based on the present location will be in error. Errors may occur in a location based service in multiple ways. For example, if the location based service provides to a user a target location of a nearby restaurant within easy commuting distance of the user's mobile device but the information is in error, there can be multiple causes for the error. A mobile device can include a traditional mobile and cell phones and other devices including smart phones, multifunction devices, dedicated devices (e.g., music and video players) wireless PDAs and computers (including handheld, palm held, laptop, desktop, wearable, etc.) and other electronic content storage and delivery devices and media.

[0055] One such cause for error in a location service that provides a target location with respect to a present location of the user wireless device is that the data representing the present location of the target, in the above case, a restaurant, can be in error. That is, the database containing the location of the target restaurant could be in error in that the restaurant is correlated with incorrect location data as read by the database having the restaurant location data stored thereon. A second source of error could be that the cellular-based location determination equipment could have inaccurate location data resulting in an incorrect present location of the user equipment. An example of this type of error could be that the cell-tower location information, stored in a database, is in error. If so, present location determination of a cell phone or other mobile device would, become inaccurate if the location determination equipment used the erred cell tower location as a basis for triangulation location of the cellular phone or other mobile device. A third source of error for cellular-based location services is in the actual determination of the resent location using the information provided. For example, if the location based-services use time differential of arrival (TDOA), angle of arrival (AOA), or triangulation principles to determine a user mobile unit location, then the calculation itself may be in error, which provides false present location determination information. Another example of error is in the use of assisted global positioning system (AGPS) information. If AGPS is used, the GPS spacecraft ephemeris is provided to a GPS receiver/processor in the mobile device which receives the spacecraft identifiers. The AGPS cell system provides ephemeris and other reference information about the GPS satellites and ground position directly to the mobile device via the cellular network so that the GPS receiver processor can more quickly calculate position information at the mobile device instead of waiting many minutes for ephemeris information to be down-linked from the GPS spacecraft. Thus, if the ephemeris, reference altitude, or reference position information supplied by the cellular system is in error, so will be the present location information derived from that information.

[0056] The user of the mobile device that receives location-based services, such as the location of nearby target points of interest (restaurants, hotels, gas stations, etc.) generally knows if the target location given to the mobile device is in error. This becomes evident if the user takes the mobile device to the suggested target location, and the target item of interest is not at the expected target location. Thus, the user of the mobile device is in a unique position to first determine if the cellular-system-provided target location information is incorrect and, if so, provide feedback on the quality of the received target location information.

[0057] In one aspect of location based services quality assessment, the mobile device user can provide user-feedback to validate the accuracy of target location-based services in a cellular network. After utilizing a location based service, such as a search for a nearby target point of interest using a mobile device, the user is provided user-equipment based means to comment upon the accuracy of the location-based service. Various implementations are envisioned. In one embodiment, the user can provide feedback via the mobile device onto which the location-based service was provided. In another embodiment, the user can use separate means, such as the internet, to provide user feedback, or a combination of both means. Other embodiments include the user answering questions via a network interface, sending and/or reply to an SMS message indicating the quality of the location-based service, the user can make a packet switched and/or circuit switched call to an automated attendant to provide information pertaining to the quality of the location based service target location data or a combination thereof. The user's response can be stored in a database or the like, for further analysis and comparison with other users' feedback.

[0058] In one aspect of location based services quality assessment, the location service quality assessment does not have to compare an absolute location accuracy with the location information provided by the mobile device. In general, correlating the mobile device user's observations of acceptable target location accuracy is sufficient to determine if the location-based services offered by the telephone network provider has become de-valued because of target location accuracy problems. For example, a measured inaccuracy of 30 feet may be of little consequence to a user and a user may not wish to report such an error. Indeed, a user may not even consider this level of target location accuracy to be a problem. But a target location accuracy error equivalent to a city block length or greater is most likely of great significance to a user. The provision of present location (spatial) and time (temporal) information further increases the value of the collected target location accuracy information. For example, in the case of wireless services, knowing where and when poor or good location service was received can help correlate the user reported data with other infrastructure data such as outage or geographic coverage issues and reports from other users. Thus, the use of accurate time (day/hour/minute) and a location service quality assessment can enhance the subjective location accuracy assessment value.

[0059] In application, a subjective location service quality assessment system may be applied to a variety of system applications. Example systems include conventional wired systems having a LAN or WAN connection, such as a voice over IP system, a terrestrial telephone system, a point to point system, and a theater system. Wireless applications for the location service quality assessment discussed herein may include terrestrial cellular, general RF communications systems, and satellite systems.

[0060] One such embodiment is in a wireless telephone network having wireless user equipment which can be a mobile telephone, a PDA, a laptop computer, or other wireless end-user equipment having location-based services capability. FIG. 11 depicts an example system 1100 useful to subjectively assess location accuracy. In one typical scenario, a user operating the user equipment 1106, requests a location-based service via the network 1104. The location-based service is typically a service offered via the network to subscribers. Location-based services include queries for the location of items of interest nearby the present location of the user equipment. For example, such requests can include queries for the locations of banks, schools, churches, shops, businesses, gas stations, town centers, bazaars, flea markets, recreational areas, scenic areas, wireless centers (hot spots), and the like. A typical wireless location-based service may accept a query from a wireless device user for a target location. The request is received by the wireless network and routed to a location-based services server(s) 1102. The query is answered by determining the location of the wireless device producing the request. This is performed using location determination equipment as part of the location based services server(s) 1102 or may be determined using present location determination equipment in the user's wireless device. Whatever means is used, the present location information is provided to the location-based service and a location of the target item of the request is sent to the user wireless equipment 1106 for display 1106A.

[0061] In other embodiments, the network 1104 may be either a wireless network, such as a cellular telephone network or other type of digital services network supporting such user equipment such as an e-mail devices, PDAs, and laptop computers. The network 1104 may also be a wired network such as a point to point system, a LAN or a WAN. At the user equipment 1106, the location based service information is received and displayed 1106A to the user. An example service would be to locate nearby gas stations for a traveler requesting such information. In this instance, the estimated present location of the mobile device is determined by the network and its present location estimation equipment 1102. Based on the estimate of the user equipment by the location based service equipment, the locations of nearby gas stations can be provided to the user equipment and displayed 1106A.

[0062] In this scenario, if the target location of the nearby gas stations is correctly provided to the user equipment, then the user would simply go to the nearest fueling station, and re-fuel the vehicle to continue her travels. However, if the location information of the nearby gas stations is incorrect or inaccurate, the user has an option to report the inaccurate location information provided to her. As indicated above, correlation of a location inaccuracy with other information is useful to aid systems engineers and administrators in locating the cause for the errant target location information.

[0063] The system 1100 permits a user to provide subjective feedback concerning the location information displayed 1106A on the user equipment 1106. The subjective user input 1106B, also called subjective user feedback, may be provided in a number of different ways. In one embodiment, a manual evaluation mode of subjective feedback may be used. In this mode, a user would construct a message that is sent to the user data feedback data store 1116. In the fully manual mode, a user may input the time, and conditions of the observed location inaccuracy as text entered through the keypad of the user equipment 1106. Another embodiment in manual mode is to record a voice message having the same information as the text message. The voice message would be sent to the user feedback store 1116 for evaluation.

[0064] In another embodiment, the user equipment 1106 may solicit user subjective information using an automated mode via the evaluation input function 1106B. In the automatic mode, the user may select a menu option while evaluating the location information. In this mode the local time may be automatically entered using automated user equipment 1106 resources such as an internal clock for time-tagging the feedback information.

[0065] In one embodiment, where the user equipment 1106 is a wireless mobile device 200 as shown in FIG. 2, the quality reporting application 222 provides the interface for the manual and automatic modes of the subjective user input to the user equipment 1106. In general, the process of collecting and analyzing subjective user feedback comports with the method of FIG. 4A. In the specific example of the acquisition of user feedback during the time that location information is being assessed by the user, the flow of FIG. 4B may be employed. In the specific instance when an auto callback is used to collect the user subjective information, the flow of FIG. 5 or 6 may be used. Post call questions may also be employed to collect user information and the basic flow for that user interface may be conducted as per the flows of FIGS. 7 and 8.

[0066] Returning to FIG. 11, in one embodiment, time information may be available as well as some local or present location information. Present location information may take the form of a GPS receiver processor or an AGPS location mechanism. In accordance with the types of errors listed above, even precise location information, when combined with faulty database information, can result in a poor quality location based service such as the location of a desired store, shop, park, and the like. Thus, as an aspect of location based services quality assessment, the user equipment 1106 may also contain a time and/or location data source 1106C as a resource for accurate time and location information. If implemented within the user equipment, the time function 1106C can provide a time tag for subjective information input by the user. For example, in one scenario, where a user is evaluating search results for a restaurant, the user may, in real time, press menu keys on the user equipment to rate the quality of the location accuracy. The user may input a 5 for excellent quality and a 1 for very poor quality. As the user actuates a soft key or other input device, a time function 1106C preferably located within the user equipment 1106 time tags the information so that the subjective rating information may be correlated temporally. As part of the quality evaluation process, the user may make multiple requests for a location based service. After or during each request, a quality assessment of the location service accuracy may be made. In this manner, the wireless network support single or multiple queries for a location-based service and the same wireless network supports the provision of feedback on the service from a user wireless device.

[0067] Considering the location function 1106C, it may reside within the user equipment or outside of the user equipment. Location determination may be performed in a variety of ways. In one embodiment, if the location function resides within the user device, a global positioning system (GPS) receiver/processor may be built into the user device such that location information is provided and tagged to one or more user subjective rating inputs. Differential GPS (DGPS) and Assisted GPS (AGPS) schemes may be implemented. Other satellite-based systems may also provide location information as well. In another embodiment within the user equipment, the location function may be such that the user is prompted to enter present location information manually onto the user equipment. The user-entered present location information, such as a street name, for example, may then be sent out along with the subjective user rating to assist in the location of the user device.

[0068] The location function 1106C may also be located outside of the user device and be part of the wireless network. An example of a location function attached to the user network is a time difference of arrival (TDOA) calculation scheme. Another implementation is an angle of arrival (AOA) scheme. Those of skill in the art will recognize that terrestrial-based locations mechanisms such as TDOA and AOA location schemes can provide the location of the user equipment using measurements from multiple base stations. This location information can be used to augment the subjective rating input information from the user concerning the quality of location based services received on the user equipment.

[0069] Subjective user feedback may be sent back through the network 1104 to be received and stored on a database 1116 on the network 1104 which accumulates the user feedback data for storage or buffering. The storage 1116 may be local or remote storage component as part of a data acquisition server, or may be implemented as an integral part of a base station. The user feedback information is accessed from storage 1116 by a data correlator 1114. The data correlator can associate the user feedback information with other network performance information to provide useful event and condition information of the network 1104. In one embodiment, the data correlator is part of the quality reporting, processing and analysis block 304 of the quality monitoring system 122 shown in FIG. 3.

[0070] In on aspect of location based services quality assessment, network performance data is acquired by observing network events and conditions. In one example, a network monitor 1108 collects network performance in a wireless network 1104. Network events such as call setups, teardowns, dropped packets and calls, switching failures, and outages are collected using the network performance monitor 1108. Also collected are performance parameters and conditions such as bandwidth utilization, packet loss, call volume and call traffic handling, data link routing and switching element performance and switching configurations for base station and cell tower configurations as well as capacity information. Such network events and conditions represent the performance, capacity, and configuration of the network 1104. The network performance and capacity data is recorded using a storage device 1110.

[0071] The data correlator can input the user feedback information 1116 with the network performance data from database 1110. By performing a correlation between the subjective user feedback and the performance of the network, an association can be made whereby a cause and effect may be identified. For example, if a network switch lost power or otherwise failed during transmission of location information to a user, then the user may experience a loss of signal or corruption of data at that time. By correlating the failure event to the subjective experience data provided by the user, a cause (the switch power failure) and the effect (flawed data) can be identified. In another instance, a change in the network performance, such as when a marked increase in call volume occurred for a base station at the same time as a data request, and a mis-read of location based data. Such a correlation can provide insight to a system administrator that a network traffic routing adjustment may be needed for a group of base stations at a specific time of day.

[0072] Correlated information can be useful to determine the cause and effect of problems that cover a wide geographic area during by identifying a common element such as latency on a server or switching or routing element, problems that correlate to a specific area (e.g., a single cell site sector) indicating a coverage, capacity or perhaps interference issue, problems that correlate to specific types of video content, or problems that correlate to specific types of handsets or combinations of handsets. Thus, the value of correlating network performance data with subjective video quality data becomes apparent.

[0073] In addition to the network performance data 1108 and the user feedback 1116, the data correlator 1114 can access other information that is useful for an assessment of location information displayed at the user device. In one embodiment, location based service information 1102 server performance and capacity data is recorded on a storage device 1112 which stores performance information concerning the location based services server. In one scenario, if the server has a fault, such as an inability to retain proper response time due to network faults or high traffic demands, then the performance of the location based services 1102 server could be adversely affected which would result in poor or disrupted service at the user device 1106. Under such conditions, knowledge of the location based services 1102 server conditions would be useful. The data correlator 1114 can also access storage device 1112 to obtain the location based services server performance and capacity information for correlation with the subjective information available in storage device 1116.

[0074] Another source of information that is useful to augment the user feedback data and the network performance data is the road traffic and weather data available via database 1118. Here, as can be appreciated by one of skill in the art, weather aberrations can cause perturbations in performance of user equipment 1106. Thus, data correlator 1114 can access the weather and road traffic data and associate the local weather conditions between a relevant base station and the user equipment to determine if the atmospheric conditions are affecting reception at the user equipment. Road traffic information may like wise be useful because road congestion may be an external source of wireless system usage. For example, given a congested section of roadway, cell phone usage may peak and aggregate to one or two cell towers or base stations. To accommodate the above-normal call volume, bandwidth available for each call being routed through those base stations, and thus a location-based service performance may be compromised. Knowledge of such road traffic conditions can thus support an analysis of degraded location service.

[0075] Another source of information useful to correlate with user feedback data is call data record information available at database 1120. Call data record information contains information as to what communications links were established, broken down, or dropped by the user equipment 1106. When accessed, call data record information 1120 can be associated by the data correlator 1114 in order to provide an event reference for the location service quality rating information with respect to the specific user equipment.

[0076] The data correlator 1114 can access information from the user feedback database 1116 and correlate the subjective information with information from a variety of databases; the network performance database 1110, the server performance database 1112, the weather and road traffic database 1118, and the call data record database 1120 along with time and location information. All or any of this databases may be accessed by the data correlator 1114. The data correlator can then associate the subjective location service quality rating from the user with time and location information, network and location server performance information, call records, and environmental condition data and identify relevant perturbations which can have a cause and effect upon the received location service quality presented by the user equipment to the user. These results can then be reported to a system administrator via report or in graphical display form 1122. The system administrator can then take the results and determine if the location service quality data can be improved by system hardware or software adjustments, or take some other action. Reports concerning the location-based service accuracy and quality assessment can be generated at 1102 to assist the administrator or system engineer in both troubleshooting and fixing unacceptable location-based service errors reported by wireless device users.

[0077] Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” Additionally, the words “herein,” “above,” “below” and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. When the claims use the word “or” in reference to a list of two or more items, that word covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0078] The above detailed description of embodiments of location based services quality assessment is not intended to be exhaustive or to limit location based services quality assessment to the precise form disclosed above. While specific embodiments of, and examples for, location based services quality assessment are described above for illustrative purposes, various equivalent modifications are possible within the scope of location based services quality assessment, as those skilled in the relevant art will recognize. For example, while processes or blocks are presented in a given order, alternative embodiments may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, or may be performed at different times. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number, respectively.

[0079] The teachings of location based services quality assessment provided herein can be applied to other systems, not necessarily the system described herein. The elements and acts of the various embodiments described above can be combined to provide further embodiments. Aspects of location based services quality assessment can be modified, if necessary, to employ the systems, functions, and concepts of the various related technologies to provide yet further embodiments of location based services quality assessment.

[0080] These and other changes can be made to location based services quality assessment in light of the above Detailed Description. While the above description details certain embodiments of location based services quality assessment and describes the best mode contemplated, no matter how detailed the above appears in text, location based services quality assessment can be practiced in many ways. As noted above, particular terminology used when describing certain features or aspects of location based services quality assessment should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of location based services quality assessment with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit location based services quality assessment to the specific embodiments disclosed in the specification, unless the above Detailed Description section explicitly defines such terms. Accordingly, the actual scope of location based services quality assessment encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing location based services quality assessment under the claims.

[0081] While certain aspects of location based services quality assessment are presented below in certain claim forms, the inventors contemplate the various aspects of location based services quality assessment in any number of claim forms. For example, while only one aspect of location based services quality assessment is recited as embodied in a computer-readable medium, other aspects may likewise be embodied in a computer-readable medium. Accordingly, the inventors reserve the right to add additional claims after filing the application to pursue such additional claim forms for other aspects of location based services quality assessment.