System for evaluating communication network services

[0002] This application claims the benefits of U.S. Provisional Application No. 60/040,475, filed Mar. 12, 1997 and entitled: METHOD OF EVALUATING WIRELESS NETWORK SERVICES

[0003] 1. Field of the Invention

[0004] The present invention pertains to systems and methods for testing communication networks or systems, and more particularly, it pertains to methods and systems for testing the service quality of various communication networks in a side-by-side comparison fashion.

[0005] 2. Description of the Prior Art

[0006] The world of communication services, and particularly wireless services, is at a strategic point. Heretofore, in addition to the conventional public switched telephone network (PSTN), there have generally been only two “cellular” service providers in any given market. With the arrival of personal communication services (PCS) and enhanced specialized mobile radio (ESMR) networks along with the various digital upgrades undertaken by the cellular incumbents, the choice of wireless network services in almost all markets is rapidly increasing and can be expected to increase by a factor of four or more in the very near future. The addition of low earth orbiting satellite (LEOS) networks and the advent of two-way paging networks (narrowband PCS) further increases the number of wireless alternatives in the marketplace. The network implementation of the technology in a given service area is one of the most important factors in creating a high quality service for the subscriber. The same technology can be implemented in a variety of ways (e.g., number and location of cell sites, frequency plans, handoff and threshold settings, etc.) to offer vastly different experiences to the end user. In most large cities, the existing cellular networks offer decent indoor and outdoor coverage but suffer from capacity (busy signals) and interference (static, co-channel clutter, dropped calls, etc.) effects. The latter shortcomings are accentuated by the segmentation of the cellular spectrum to accommodate digital services. With smaller subscriber bases, the newer PCS services do not generally suffer from traffic related impairments; however, obtaining locations and erecting antennas for cell sites is getting to be increasingly difficult for a variety of reasons which results in potentially poorer coverage for the newer entrants.

[0007] An evaluation problem immediately presents itself when considering network quality, which problem is accentuated by the proliferation of different types of communication services as pointed out above. With all of the various factors that affect cellular service and particularly the quality thereof, a problem has arisen in being able to compare the network service attributes in an objective manner in order to properly rate them so that one can make an informed choice of provider and type of service for a particular area of usage. It may also be critical to compare the various wireless services with conventional (landline) communication networks in order to properly evaluate the quality of the service provided.

[0008] In the past, various systems and methods have been used for the testing of a specific communication network service, particularly a wireless service, over a varied geographic area. These systems generally were designed to provide data on a number of engineering parameters such as signal strength, etc. For example, U.S. Pat. No. 4,977,399 to Price et al discloses a system which uses a mobile unit, i.e., a van, which includes means for simultaneously receiving radio signals including test data from a plurality of different transmitters and a computer for processing the received test data from the radio transmitters to directly compare the results. As the mobile unit travels, its position is simultaneously recorded by means of a LORAN antenna so that the geographical position information can be directly related to the data transmission. The system is thus able to provide information relating to the range and quality of the radio paging signals as they are received at various sites from a particular mobile station travelling over the varied geographic area.

[0009] U.S. Pat. No. 5,023,900 to Tayloe et al discloses a system and method of evaluating the cellular radio coverage of a geographic area serviced by a digital cellular radio telephone communications system. A mobile unit transmits data to a plurality of base stations of the cellular radio telephone communication system within the geographic area and simultaneously transmits information with regard to its specific (longitude, latitude) location. Each of the base stations transmits the information to a central operation maintenance and control unit which includes a computer that processes such information to accurately determine the performance of the cellular phone system as the phone is moved about the particular geographic area under testing.

[0010] A further wireless testing method and system is disclosed in U.S. Pat. No. 5,398,276 to Lemke et al which discloses a portable cellular telephone signal strength analyzer which includes a laser range finder with a built-in compass to input data on the geographic locations of the analyzer as it is moved about a specific geographic area, e.g., a shopping mall or office building.

[0011] Other United States patents which disclose testing systems for wireless networks and particularly cellular networks include the patent to Knippelmier U.S. Pat. No. 5,425,076 which discloses a measurement module located at a base station that initiates a call to one or more responders at remote locations and monitors the quality of the calls and receptions with a wide variety tests of both engineering parameters and service quality; the patent to Yehushua et al U.S. Pat. No. 5,504,800 which discloses a system for testing cellular phones that includes a series of transmitted and received messages generated in response to each other and which includes a computer which provides a quality evaluation of the transmission/reception; the patent to Siu et al U.S. Pat. No. 5,528,661 which discloses a self-diagnostic test routine for use in generating test signals in remote measurement units which communicate with various subscriber lines in the vicinity and wherein the testing equipment at the remote measuring units are checked periodically or continuously against various preset levels to determine the status thereof; the patent to Hong et al 5,539,804 which discloses a system for monitoring call quality over a plurality of transmission lines with such call quality including signal-and-noise ratios, speech level, noise level, echo characteristics, etc., and which system uses a central computer operating through a public switched telephone network to receive calls from the various transmission lines; and the patent to Kuenzig 5,572,570 which discloses a method of testing a telecommunication system by a sequence of voice and non-voice signals by comparing the responses from the tested systems with predetermined standards.

[0012] With the method and system of the present invention, there is provided a system which can directly compare in side-by-side comparison fashion different types of network communication services or different providers within a single type of network communication service as, for example, a plurality of cellular phone services within a given service area. The testing routines are carried out and the test results are provided with the end user in mind.

[0013] Thus, a means is provided at a first location, which may be a mobile vehicle continuously moving through a geographic service area, for sending a first set of identical preselected messages to a second, fixed location through the plurality of network communication services being tested. At the second location a second set of identical preselected messages are sent back to the first location with each of the messages of the second set being sent directly in response to the corresponding received first message. Then, means are provided at each location for quality grading each of the messages received and for providing a score with a comparison of the scores between the various services providing a comparative quality grading for the different communication network services or providers tested.

[0014] Additionally, when wireless services are being evaluated and a mobile vehicle is used as the first location, geographic position and vehicle speed information may be correlated to each message in order to better evaluate the different wireless networks throughout the varied terrain of the geographic area under test.

[0015] FIG. 1 is a diagrammatic illustration of a wireless network evaluating system utilizing the present invention.

[0016] FIG. 2 is a block diagram illustration of the components of the system of FIG. 1 at the mobile end thereof.

[0017] FIG. 3 is a block diagram illustration of the components of the system of FIG. 1 at the fixed end thereof.

[0018] FIG. 4 is a diagrammatic side elevation, partially broken away, of the mobile unit in the system of FIG. 1, particularly illustrating the manner in which the various cellular phones to be evaluated are positioned.

[0019] FIG. 5 is a flow chart particularly depicting the computer programming for the subroutine of initiating the calling during a test sequence of the evaluation process.

[0020] FIG. 6 is a flow chart particularly depicting the computer programming at the calling end during the test sequence.

[0021] FIG. 7 is a flow chart particularly depicting the computer programming at the called end during the test sequence.

[0022] FIG. 8 is a flow chart particularly depicting the computer programming for the subroutine of recording time, position and vehicle speed during the test sequence.

[0023] FIG. 9 is a flow chart particularly depicting the computer programming for the subroutine which determines the call quality after the test sequences.

[0024] FIG. 10 is a block diagram illustration of the basic components of a second system for evaluating a plurality of different communication network services in accordance with the present invention.

[0025] While the system of the present invention is particularly designed to evaluate, on a side-by-side comparison basis, communication network services of various types, including particularly wireless network services, in the particular embodiment of the invention shown and described herein, the communication network services will consist of a plurality of wireless cellular networks all operating within a given geographic area.

[0026] The general arrangement of the system of the present invention for performing a wireless network evaluating method, particularly for competing cellular phone service providers, is schematically shown in FIG. 1. One or more mobile units 10 are provided which drive about the various changing terrain of the area to be evaluated with each mobile unit being provided with a plurality of individual phones 12 (FIG. 4) each such phone being adapted to communicate with one of the particular networks to be evaluated. As shown in. the example of FIG. 1, four separate networks are tested, e.g., networks A.B,C and D. As the mobile unit 10 moves about the general calling area to be monitored, calls are made from the cellular phones carried in this unit by simultaneous radio transmissions to the various radio towers associated with the network services A-D from which towers the calls can proceed in conventional fashion via the conventional public switched phone network 14 to a single, fixed-end unit 16 which has been provided at some appropriate location, preferably, but not necessarily, within the area of evaluation. Both the mobile unit 10 and the fixed-end unit 16 include computers (CPU) and appropriate software for initiating calls, for receiving calls, and for evaluating the quality of the individual segments of information (voice or data) transmitted during the calls by means of conventional voice recognition or authentication software or conventional digital data comparison techniques. Such units 10, 16 also include the capability of storing and retaining the various data accumulated during the calls and for making overall quality evaluations based on the individual quality evaluations of the separate voice or data transmission segments.

[0027] The layout of the equipment at the mobile unit 10 is shown in FIG. 2 and is schematically illustrated in FIG. 4. Thus, as explained above, each of the phones 12 in the mobile unit is particularly adapted to be used with a particular cellular network which is to be evaluated. As shown in FIG. 4, the phones are mounted on a common plane at a level approximately that of the vehicle driver's head where they would normally be used during the making of a call from an automotive vehicle. As can further be seen in FIG. 4, the phones may be mounted upon the upper periphery of an upright cylindrical turret structure 13 that is adapted to sit upon the transceiver interface hardware 21, the computer or central processing unit (CPU) 20, and the power supply 22 located centrally within the vehicle. A connecting line 18 from the transceiver interface 21 to a suitable switchbox 19 on the vehicle dashboard may be provided so that the calling or transmission sequence can be initiated by the vehicle driver.

[0028] As can be seen in the block diagram illustration of FIG. 2, the transceiver interface hardware 21 at the mobile unit 10 is comprised of various hardware components that are interfaced between the phones 12 and the computer 20 and which include a conventional voice card or pc-telephony card 23 which is adapted to digitize the analog signals from the phones for transmission to the computer and to convert the various digital information from the computer to the appropriate analog voice signals for transmission over the phone networks. The interface components also include a conventional modem 24 to send and receive the various data files and packets that may be transmitted over the phone lines in addition to the voice recordings, and a conventional global positioning satellite (GPS) receiver 26. A special phone interface unit 12a is provided for each of the phones so that the signals from the voice card 23 can be matched with the particular input jack on each individual phone 12.

[0029] At the central or fixed-end location 16, as shown in FIG. 3, a somewhat similar arrangement of components is provided with a computer 30 operating through a conventional voice or pc telephony card 27 (similar to the voice card 23) for interfacing with the local telephone system during voice transmission and a modem 28 (similar to the modem 24) for interfacing with the telephone system during the transmission and reception of data files and packets. A conventional telephone interface arrangement or switchboard 32 is also provided which will be equipped to handle simultaneous calls from all of the mobile phones being tested, and which switchboard is linked in the customary manner to the local public telephone network for receiving and transmitting calls.

[0030] The various wireless network telephones to be used in the aforedescribed system are purchased through authorized retail channels and are then integrated without modification into the automated test apparatus as previously pointed out. Clocks within the CPU at the mobile end 10 and fixed-end 16 are synchronized so that the transmission of information therebetween can be appropriately automatically correlated. The GPS receiver 26 at the mobile end 10 (FIG. 2) is set to provide updated vehicle geographic location (latitude, longitude) information as well as vehicle speed information at fixed time intervals (about every four seconds at present) which will be maintained in a separate file and then input into the system during the analysis of the accumulated data so that each individual data segment or voice transmission segment can be accurately tagged as to the particular vehicle location and speed.

[0031] The testing process of the present invention is computer controlled and consists of a series of calls each of which can be originated from either the fixed-end 16 or the mobile end 10 in a preselected sequence. Each call will consist of an alternating transmission and reception of information (voice or data), and both the sequencing and the content of each call will be identified for each of the network services A-D being tested. It is an important feature of the present invention that each of these calls is initiated on each of the wireless networks under test at exactly the same time; thereafter, the progress of each individual call is separately monitored and clocked so that the results thereof can be compared. While the calls may vary in duration, they are typically of about two minutes in length. The calls may consist of a voice recording, wherein the calling end randomly picks a sentence from a predetermined vocabulary of sentences with such vocabulary consisting of prerecorded sentences spoken by a variety of speakers and taken from an industry standard such as the diagnostic acceptability measure (DAM) or Harvard sentence sets. Also standard digital data transmission in files or packets may be utilized as a separate call consisting only of the transmission of data. The calls are placed throughout the region of interest at all times of the day with, in a typical testing sequence, about five calls being placed per square mile within the area to be tested. The calls can be originated from either the mobile end or the fixed end, and it is important that both options be used in order to fully test the mobile telephone's capabilities.

[0032] Flow charts indicating the programming for the computers 20, 30 at the mobile end 10 and fixed end 16, respectively, are shown in FIGS. 5-9. Thus, with reference to FIG. 5, upon the random initiation of a call at the mobile end 10, the mobile end computer 20 will immediately initiate the record location process (see FIG. 8), i.e., initiate its clock and the GPS receiver 26 to continuously update the vehicle location and speed information at regular intervals. If the initiation of a call is at the fixed end 16, the record location process will be initiated at the mobile end immediately upon the reception of the first message on one of its phones 12. Each computer 20, 30 then determines from the source of the call initiation signal (self-initiated or call response) whether it is the calling end or the called end, and the sequence of steps as set forth in FIGS. 6 and 7, respectively, is then put into operation.

[0033] The calling end process for voice transmission is set forth in the flow chart of FIG. 6. The process will first determine that all of the lines or phones 12 (M=the total number of networks under test) are free for use. The random selection of the sentence to be sent is then made, and a call is simultaneously placed on each of the lines or by each of the phones with the start time thereof being recorded. From that point on each individual call on an individual network is separately monitored as to time with each call segment or sentence being tagged with updated time information. When the call is answered, i.e., an answering call segment sent back to the call initiator, the time information is again recorded, and a series of sync tones (generated in the computer and converted in the voice card 23 for voice transmission) are sent both by the calling end and the called end in order to synchronize the automatic alternating transmission and reception of call segments or messages (e.g., sentences) therebetween during the subsequent duration of the call. Upon the appropriate reception of sync tones back from the called end within a given time interval (T2 seconds), a sync tone is sent to the called end and the transmission of the sequence of voice sentences is initiated, and in the calling sequence set forth, each transmission of a sentence (after the first one) is interspersed between a received sentence from the called end. The transmission quality of each received sentence is scored by conventional voice recognition software to be explained more fully hereinafter. At the end of the calling sequence, i.e., after a predetermined number N of sentences are transmitted, the telephone connection is terminated with the time and call duration being noted. In a manner to be further described, the call quality is then evaluated. As can also be seen from the flow chart of FIG. 6, various circumstances which may arise during the calling sequence will automatically terminate the sequence. For example, if the call fails to connect (i.e., be answered) this is recorded and the reason for such failure to connect (if ascertainable) is similarly recorded, e.g., if the problem is coverage (the wireless phone system is unable to process the call) or blocking (the network is too busy to process the cell). Also, if the transmission and reception of the synchronization tones are not completed within the prescribed time limit T2 so as to initiate the transmission/reception sequence, or if the call ends before its prescribed completion time, such event is similarly recorded.

[0034] The computer at the called end goes through a similar sequence of steps for voice transmission as set forth in FIG. 7. As can be seen, prior to the reception of an incoming call, a sentence is randomly selected and ready for transmission. Then, upon the reception of an incoming call and after recording the time thereof a sync tone is sent to the calling end in order to synchronize the sentence transmission. When the response sync tone is received from the caller within the prescribed time limit T3 the called end then transmits its sentences at prescribed intervals which in accordance with the predetermined timing sequence fit alternately between the prescribed time intervals of the calling end sentences. As with the FIG. 6 calling end process, each received sentence is individually scored for quality, and the overall call quality is also scored at the end of the calling sequence. Also, as with the calling end program sequence, those events which cause the call to be terminated (e.g., no synchronization, call cut off before its prescribed end, etc.) are recorded along with the appropriate timing information.

[0035] The subroutine for the record location process performed at the mobile end and in the subroutine illustrated in FIG. 5 is set forth in FIG. 8. It will be seen that the system immediately records the time, X, upon initiation of the call and that such time information is updated every T seconds, with T being about four seconds as the system is used at present. At each time update, the vehicle position (longitude and latitude) and speed are also input into the system as indicated. This information is maintained in a table in the computer, and it is used when the call quality process is initiated so that each call segment, i.e., sentence, will be tagged with the correct mobile end location and vehicle speed.

[0036] The subroutine for determining the call quality process is set forth in FIG. 9. This process can be performed using conventional voice recognition or voice authentication software, and in the process as shown, all of the voice recognition is done within the computer software. Each received sentence (J, K, etc.) is compared with a template of the expected sentence and a matching score (S1 to SN) is generated using standard speech recognition and verification algorithms with a higher score implying better quality. Then, from each series of matched scores representing the sentences received during a single call the maximum (Smax) and the minimum (Smin) are determined. Also, an average score for each sequence (Savg) is determined and the standard deviation from the average (Sstd) is determined. This information may then be used to make a final evaluation (good or bad) on each call. For example, if Smin is more than some predetermined quality (C1), and/or if the ratio of Smax to Smin is less than some predetermined quality factor (C2) the call may be labelled as “good”. Finally, the ratio of the standard deviation (Sstd) to the average quality (Savg) is determined, and/or if this ratio is less than a predetermined threshold value C3 the call may similarly be labeled as “good”.

[0037] While the foregoing sets forth the transmitting sequence during voice transmission, a similar sequence of steps is utilized during data transmission. For example, predetermined selected digital data files may be transmitted in either direction and the file transfer time recorded. Transmission integrity is determined by comparing the received file with the transmitted file using conventional digital comparison techniques (algorithm not shown) and recording the number of bit errors. The data throughput is determined from the file transfer time and the file size. Also, short bursty transaction oriented packets may be transmitted in either direction to bench mark data system performance for transaction oriented applications.

[0038] At the end of the test, data from the fixed and mobile ends is merged based on the time tags thereon so that location to location within the geographic area can be directly compared for each network M being tested. Post analysis of the data can provide a variety of measures for absolute and relative service quality including the following factors: (1) Access—does the call go through on the first attempt, and if not, is it because of lack of coverage or blocking?; (2) connect times—how long does it take to make the connection?; (3) retention—does the call hold through the conversation or does it unexpectedly terminate in midsentence?; (4) quality—is the quality of the call good or bad?; (5) data set-up times—how long does it take to set up a data session?; (6) throughput—what is the data throughput in either direction?; (7) transit delays—what is the one-way and two-way transit delay through the network?

[0039] While the foregoing system and method shows a plurality of wireless network systems being tested with the cellular phone end of the network system being located in a mobile vehicle, it is possible to use the same testing process, particularly as set forth in FIGS. 5-9 (but without the GPS unit and any mobile/fixed distinction), to directly compare a plurality of entirely different types of communication network services between a pair of fixed locations. Thus, in FIG. 10, there is shown a system for transmitting information between two fixed locations 41 and 42, each of which contains hardware and software similar to the fixed location arrangement 16 previously described. With such an arrangement, simultaneous side-by-side testing can be made through the local public switched telephone network (PSTN) 44, the internet 46, a PSTN hybrid system using satellite transmitters 48, various local cellular providers 50, and wireless local loop (WLL) systems 52. In this fashion, the various wireless services can be directly compared with the current service standard, the PSTN, for a relative quality grading.

[0040] Although the best modes contemplated for carrying out the present invention have been herein shown and described, it will be apparent that modification and variations may be made without departing from what is regarded to be the subject matter of the invention.