Product development and assessment system

[0001] This application claims the benefit of priority to the following provisional applications, the respective contents of which are hereby incorporated by reference in their entireties: U.S. Patent Application No. 60/462,800, filed Apr. 14, 2003; U.S. Patent Application No. 60/463,699, filed Apr. 17, 2003; U.S. Patent Application No. 60/515,962, Oct. 31, 2003.

[0002] A. Field of the Invention

[0003] The present invention is directed to a product development and assessment system and method. Specifically the invention is directed to a method, system, and program product for conducting network-based surveys of respondents (such as prospective or existing consumers or customers) using continuous metric scales; developing a framework within which respondents rate new or existing products, services or advertisements; translation of the survey results into a readily understandable score, and automatic interpretation and reporting of the data and score in plain English.

[0004] B. Background

[0005] Traditionally, product and service development involves evaluation of customer acceptance. During economic slowdowns, companies typically have fewer resources to devote to time and cost intensive customer studies. A rapid, reliable and inexpensive system for getting large scale consumer evaluations of new product concepts, and developing these concepts into full implementation and execution would be desirable.

[0006] Furthermore, both venture capitalists and investors capitalized Internet start-ups at historic levels—even though the start-ups often had little marketplace experience, no profits, and no measure of whether customers would adopt their product or service over the long run. The bottom dropped out of this market. Some of the companies were popular, but did not have profit models to sustain them. Other companies failed because customers were dissatisfied with either the product or the execution of the product, i.e., how easy or charming was the experience of doing business on-line. One factor leading to the funding debacle was that venture capitalists did not have the tools to assess whether consumers would try, like, and adopt the new electronic product or service. The new companies focused their capital on building expensive systems, databases, communications systems, applications and even factories and advertising before they knew whether customers really wanted the product or service as the company planned to deliver it.

[0007] Investors typically rely on business plans to determine whether or not to invest. Business plans, however, typically lack a reliable objective indicator of: whether the target customer wants the idea; whether, if consumers try it, it delivers what the idea promises; and whether the customer perceives that the electronic product or service is an effortless, charming experience that works at the customer's speed and is an improvement over the customer's old way of acquiring the product or service.

[0008] Today, many companies use focus groups to estimate customer response to new concepts, products, and advertisements. Focus groups can identify issues, anecdotes and customer language. The results, however, are often narrow. Focus group results, moreover, can readily be manipulated by the subjective nature of the questions and the moderator's position on the product, service, or advertisement. Focus groups have their use, but focus groups cannot deliver rigorous quantitative and qualitative evaluation and diagnosis of the concept, product, and advertisement.

[0009] Those skilled in the art appreciate that the most reliable indicator of the potential acceptance of a product is “rank order data.” Rank order data is a measurement of the customer's view on the relative appeal of the new product concept compared to the customer's current product. Unfortunately, obtaining valid and reliable rank order data is often difficult, costly and time-consuming. It is difficult because there is often no current method of having customers identify their preferences in an objective manner. Even if there were a current way to secure customer's preferences, researchers would require statistically significant samples to be reliable, and it would be time-consuming and expensive to acquire these samples using traditional methods.

[0010] Having identified the aforementioned problems in the existing methods of consumer surveys and product development and assessment, the inventor has developed the present invention. As described in the present application, the invention provides a method, system and program product for conducting network-based surveys of respondents (such as existing or potential consumers or customers, or other targeted respondents) using continuous metric scales and developing a framework within which respondents rate new or existing products. A continuous metric scale can be depicted as a line, preferably one without end points. When a respondent identifies a rating on the continuous line, the rating is recorded as a demarcation on the line.

[0011] The present invention discloses a method, program product and system for conducting product assessment and development, where the method comprises displaying on a terminal a predetermined scale against which a respondent can indicate a rating for a first existing product, receiving a first existing product rating indicated against the scale from the terminal of the respondent, displaying a visual indicator along the scale corresponding to the first existing product rating, receiving a first different product rating indicated against the scale from the terminal of the respondent, displaying a visual indicator along the scale corresponding to the first different product rating and collecting ratings made by the respondent via the terminal.

[0012] The invention also includes a method of conducting assessment and development of a product using information from a plurality of targeted respondents (such as potential or existing customers or consumers), where the method comprises collecting answers and ratings made by a plurality of respondents using the method described above and generating a product assessment report for the product.

[0013] In the present specification, the term “product” is defined broadly to include any method, service, or article of manufacture. The term “product” also includes all of the features and marketing components of any method, service or article of manufacture, including by way of example, but not limited to advertising, marketing plans, brochures, packaging, pricing, positioning, and other components that those skilled in the art would appreciate. Accordingly, the present invention may be used to assess methods, services, articles of manufacture, and their underlying features and marketing components, whether the “product” is pre-existing or new.

[0014] The invention described herein solves the problem described above regarding the use of business plans by providing a reliable and readily replicable objective measurement system. The instant invention also solves the problems involved in obtaining valid rank-order data by enabling respondents to identify their views on their current favorites and their view of the relative appeal of the new product all on the same continuous metric line scale. Their current favorites act as a metric anchor. In addition, the use of the web-based survey technique enables the developer to obtain statistically significant samples on a fast and cost effective basis.

[0015] Other features and advantages of the present invention will become apparent to those skilled in the art from the following detailed description. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the present invention, are given by way of illustration and not limitation. Many changes and modifications within the scope of the present invention may be made without departing from the spirit thereof, and the invention includes all such modifications.

[0021] The present invention is now described in detail in reference to certain preferred embodiments of the invention. General features of the invention are first described. Next, particular embodiments are described in reference to the above-mentioned figures.

[0022] Generally, the present invention can be summarized as a management information system and method for the development and assessment of products and services. One preferred embodiment described herein is referred to as a Linescale Product Development System. This exemplary system and method relates to a web-based survey technique that facilitates developing and testing new products, business investments, and value propositions, as well as testing existing products, businesses, and value propositions. This technique includes systematic solicitation of respondents, preferably potential customers, eliciting data from respondents, automatic processing, interpretation, and scoring of the data, and automatic report generation. Using an on-line system, this embodiment of the invention provides a fast, engaging, and inexpensive way to gauge customer acceptance and use.

[0023] The interactive development tool first guides generation of the concept, then development of the product or service and its marketing components. As explained below, the combination of the two techniques, when used iteratively, provides a unique management information system, method, and program product for product and service development, and business assessment tool.

[0024] The invention provides a behavior predictive development tool rather than an attitude measure. The invention enables respondents, including potential customers, to display their preferences for the new concept or product compared to their currently used favorites. The invention provides a systematic way to determine how target respondents would score a new product concept, how the idea stacks up against the respondent's two current favorites, how the execution of the concept or idea being tested performs compared to the concept itself and the two favorites, and which competitors or current behaviors they would switch away from to use the new product or service. The system processes and translates the data into a readily understandable and replicable score. The system provides both quantitative and qualitative development guidance based on scale ratings and respondents' verbatim reasons for making their comparative ratings. This gives precise guidance to product developers, management, and investors on probable trial, repeat and depth of repeat. The analysis shows the user who wants the product and why, how to improve it, who hates the product and why, and points to what needs to be done to fix the product.

[0025] The invention relates to a pure development tool built to help companies identify, create and build successful new products and fix old products. The iterative method of test and repeat allows companies to make bets and see if they pay off—long before going to market. The invention recognizes that fast, inexpensive iteration is the route to development success and that almost nothing works in first draft. The invention provides a system for a company to get the drafts behind them before commitment to production and introduction, in a more productive, effective and efficient manner.

[0026] The invention facilitates the assessment of existing products and businesses on a fast and cost-effective basis. It further facilitates product development and business investment decisions by providing a quantifiable prediction of customer acceptance of a new product.

[0027] As a product development tool, the invention addresses the issue of how to develop products and services on a rapid and cost-effective basis. As the economy recovers from the recent slowdown, companies need to replenish their R&D pipeline of new products. Because of the past reduction in development spending, there is a gap in the traditional development process that requires a rapid screening, development and implementation of new products geared to the consumer, professional and small business markets. The present invention fills this need by providing a rapid, reliable and inexpensive system for obtaining large scale consumer evaluation of new product concepts, and developing these concepts into full implementation and execution.

[0028] This invention relates to a system and method for guiding the concept screening and development for a new product or service designed for consumers, professionals or small business. After development of the refined concept, the invention guides the development of the product or service itself, including the method of operation, customer requirements, product or service performance characteristics and all elements of the marketing mix, including brochures, advertising in all media, direct mail offers, packaging, premises, etc. The invention allows a user to make better investment decisions in new business ventures by providing a quantitative method of determining customer acceptance of a product.

[0029] The invention achieves these results using a web-based survey technique using continuous metric scales, translation of the survey results into a readily understandable score, and automatic interpretation and reporting of the data and score in plain English.

[0030] This invention provides the following: a quantitative and objective way of predicting acceptance by a group of respondents of a product or service; a step-by-step way of eliciting anchored rank order data by using metric anchors; a readily understandable score based on a respondent's relative ranking of a product that reliably predicts acceptance of the product; an inexpensive way to obtain a large sample of respondents, through the web and iteration via the web, thus providing reliability and speed at a low cost; a way of improving a concept on a step-by-step basis; a way to determine, by the use of norms, when a concept is “good enough,” improvable, or whether to discard the idea; a way of improving a product or service derived from a concept on a step-by-step basis; an automated way to use respondents' relative ranking of items to predict acceptance and use over time; a way to predict acceptance by customers or other respondents in a way that can be conducted manually, or on a fully or partially automated basis, in person or on a remote basis through a communications network; and a way to help customers or other respondents accurately report the relative ranking of a new product by using continuous metric scales to compare the relative ranking of dissimilar products with anchors.

[0031] One embodiment of the invention facilitates two stages of product development: development of the concept by means of a Concept Developer and development of the product by the Product Developer.

[0032] The Concept Developer assesses customers' or other respondents' reaction to the concept of the new product or service. To determine customer reaction, the system elicits rank order data from the customer. It requests the customer to rank the customer's current favorites, the new concept, and a more detailed description of the concept. Then the system asks the customer to rate the idea on a metric scale for believability, uniqueness, importance to them personally, and the probability of recommendation to friends.

[0033] In one embodiment of the invention, described in greater detail below, scoring is numerical. By way of example, it can be based on six measures, such as the following:

[0034] a) 0, 5 or 10 on percent preference for concept over current product used and next favorite product used—assigning 5 points to preference of concept over each favorite (two current products)

[0035] b) 0 or 10 on percent preference of product experience over concept

[0036] c) 1 to 6 on believability

[0037] d) 1 to 6 on uniqueness

[0038] e) 1 to 6 on importance

[0039] f) 1 to 10 on will recommend it to friends

[0040] In this preferred embodiment, calculating the score involves taking a weighted average of the answers and ratings in accordance with a predetermined formula. In this embodiment, adding up the six numbers, taking the average and multiplying by 100 derives the score. Preferably, the algorithm to determine the likelihood of customer or other respondents' acceptance and trial of the new product is I=((a1+a2)+b+c+d+e+f)/6×100. Using this algorithm, a system according to the invention displays all of the relevant data and scores on one readily understandable metric scale. While one specific weighted average is described herein, those skilled in the art will appreciate that any weighted average or combination of scores could be used under the present invention.

[0041] The Product Developer assesses likely customer or other respondent reaction to the experience of the new product or service. In the product developer, the system enables the respondent to rank the current favorites, the concept, and a more detailed description of the concept. Then the system asks the respondent to rate the idea on a line scale for ease of use, perceived speed, charm or pleasantness of experience and probability of recommendation to friends.

[0042] As with the Concept Developer, scoring for the Product Developer of one embodiment preferably is numerical, and may be based, for example, on six measures:

[0043] a) 0, 5 or 10 on percent preference for concept over current product used and next favorite product used—assigning 5 points to preference of concept over each favorite (two current products)

[0044] b) 0 or 10 on percent preference of product experience over concept

[0045] c) 1 to 6 on ease

[0046] d) 1 to 6 on speed

[0047] e) 1 to 6 on charming, pleasant

[0048] f) 1 to 10 on will recommend it to friends

[0049] Adding up the six numbers, taking the average and multiplying by 100 derives the score. For the Product Developer, the preferred algorithm for determining the respondent's reaction to the experience of the product compared to his own product and the likelihood of adopting the new product is I=((a1+a2)+b+c+d+e+f)/6×100.

[0050] In the preferred embodiment, the scores for both the Concept Developer and the Product Developer provide are then assessed. For example, using the algorithms noted above, scores may be ranked as follows:

[0051]

200 to 300	Forget about it
301 to 400	Not ready for prime time
401 to 500	Something here, but it needs work
501 to 600	Good concept (or other noun), works pretty well
601 to 700	Very strong idea (or other noun), great execution
701 to 800	Customers are wild about it (or Excellent)

[0052] In exemplary embodiment of the invention, the key score is the percent of the sample scoring the concept or product over 600. In this embodiment, the percentage of customers scoring the new concept over 600 would be considered the top score using the technique according to the invention. This has a high correlation with subsequent product success in the market place.

[0053] The system generates sample thresholds or norms for product performance based on the experience of testing many similar products. The thresholds are the scores that successful products achieve. The user of the system may choose a threshold or adopt one recommended by the system for the type of product being tested.

[0054] The system is designed for multiple iterations. Iterations include testing the concept or product experience, reviewing the test results, modifying the concept or product based on the test results and repeating the tests with new survey participants. Both the concept developer and the product developer are used in iterations of the concept or product features until the user of the system achieves the desired results.

[0055] The Concept Developer is iterated until one of two events occurs: either the developer gives up, since the concept and a more detailed description of the concept cannot be improved to meet target thresholds, or having exceeded target thresholds, subsequent iterations do not improve concept performance.

[0056] The Product Developer works in a similar manner. The system asks the customer or other respondent to compare the product concept to various implementations of the product, such as on-line applications, advertisements, direct mail offerings, physical embodiments of the product, product experiences of taste, wearability, usefulness, quality, or many other examples that would be apparent to those skilled in the art. The system asks the customer to rank each component under consideration quantitatively and objectively to identify how strongly the component supports the concept. Based on the customer's ranking, the system generates an overall score and a score of how well each of the components supports the product concept. If a key component does not meet minimum thresholds, the system generates a plain English message to the user of the system. If additional strengthening, addition of elements to the product, or reduction of the price does not improve performance, the system generates important information to the developer.

[0057] The Linescale system is an effective tool for assessing advertising, packaging and other marketing components. In another embodiment of the invention referred to as an Advertising Developer, the system according to the invention is used to assess the persuasive effect of advertisements, packaging or other marketing materials. Using the same techniques as the Concept Developer and the Product Developer, the Advertiser Developer system asks the customer or other respondent to provide his current assessment of the existing or new product and the customer's two other products or relevant behaviors. For example, in testing a weight loss product, a customer would be asked to assess the weight loss product and his other methods of achieving weight loss, such as exercising, taking diet pills or reading healthy cookbooks. After providing relative rankings of the product and the customer's two favorite products or behaviors, the Advertiser Developer would display a commercial or story board to the customer. After viewing the commercial twice, the customer would be asked to re-rate his assessment of the product and identify his reasons for the rating both quantitatively and objectively. As with the Product Developer and the Concept Developer, the Advertisement Developer would generate a detailed report that would provide detailed guidance on the persuasive nature of the advertisement.

[0058] It will be appreciated that the Linescale System according to the invention is effective for assessing existing products.

[0059] Another aspect of the invention described in greater detail below is referred to herein as the Marketing Report Card. According to this aspect of the invention, the system uses the same techniques as the Concept Developer and the Product Developer to assess ongoing products. For example, in testing customer attitudes about a weight loss system, the Marketing Report Card asks customers to rate the weight loss system and the customer's two other favorite weight loss products or methods. With these competitive anchors in place, the system asks the customer to rate each of the components of a system. For example, a customer may assess aspects of a weight loss system, such as a diet center, staff, food, price, and instructional materials. After presenting the customer with a score for his ratings, the system asks the customer whether he agrees or disagrees with the assessment he has provided and the system asks the customer for quantitative and subjective reasons for the rating. The system generates a report that provides valuable feedback about what aspects of a product work and which do not in a rapid and easily understandable manner.

[0060] It will be appreciated that one aspect of the present invention that creates the reliability and stability of the data generated by the Linescale System is the way it creates the “competitive frame” or “customer space.” This means that the system helps place the respondent in the right frame of mind to answer the questions in a reliable fashion. Without establishing this framework, customers may give answers that do not provide an accurate predictor of their behavior. Without the framework, respondent answers may be influenced by whim or a typical human response to novelty, order of questions, or the difficulty of identifying true preference in the abstract.

[0061] The personalized selection of favorite brands, products or practices in the market space as competitive frame is a unique stabilizing factor as well as a benchmark for evaluation. Market research today typically may use a “control” such as a given brand or product currently offered in the market place as an “anchor”. Another common practice is to use an “ideal” imagined product as a control. But each of these lacks the personal commitment and clarity of understanding based on the respondent's personally selected items or practices as current favorites.

[0062] The method for establishing the framework is to ask the customer or other respondent to identify his or her current favorite activity or brand and then to identify his or her second favorite activity or brand. The system often provides the respondent with a wide array of activities and brands as choices to get the customer in the right space. Products and services may cross many markets, so it is often desirable to identify the various activities associated with the product to give the respondent the right framework.

[0063] It will be understood by those skilled in the art that the invention may be deployed in many different embodiments. The following describes preferred embodiments of the invention in reference to the accompanying drawings.

[0064] FIG. 1 is a block diagram showing a system according to one preferred embodiment of the system. While this exemplary system is web-based, it will be appreciated that other approaches may be utilized.

[0065] The system 1 includes a host computer 10 such as a server computer, a network 11, a user terminal 12 and an array of terminals, generally referred to by references 13-15. The terminals may comprise personal computers, or other types of terminals known in the art. Target respondents (typically comprising, for example, prospective or existing consumers or customers) of the subject product, use the terminals 13-15 to provide feedback about a product concept to the host computer 10. In the embodiment of the system discussed herein, a survey preferably is conducted with a minimum of 48 respondents, but other numbers of respondents could be used. Thus, it should be understood that while four terminals illustrated, a larger number will preferably be used.

[0066] As illustrated in the examples that follow, the host computer 10 transmits screens to the terminals 13-15 over the network 11. The host computer 10 also transmits reports to the user of the Linescale system at the user's terminal 12 over the network 11. The user may also use the user terminal 12 to access the host computer 10 to obtain the reports from the surveys, as described generally below.

[0067] FIGS. 2A-2C are flow charts showing general features of operation of a system according to one embodiment of the invention, as described in more detail below. It will be understood that steps illustrated could be repeated; further, the steps would not necessarily be follow in the specific order illustrated.

[0068] Step S10 represents a step of contacting respondents that may comprise, for example, potential or existing customers, consumers or other targeted groups of respondents, and soliciting their participation. This may be accomplished, for example, by a letter, electronic message or other communication that includes a URL to access a system that conducts the survey. Once the respondent logs onto the system, the respondent receives and provides information that is used for concept and product development and to generate reports.

[0069] In the example shown, a respondent begins a survey at S20. Preferably, the respondent provides information about the respondent's current practices and/or preferences with respect to an existing product, which as noted above, may comprise any method, service or article of manufacture, as well as all of the features and marketing components of any method, service or article of manufacture. Based on these responses, the system displays a summary of the respondent's current experiences at S30.

[0070] The system then prompts the respondent and receives ratings for the existing product at S40. According to one feature of the invention, this is done in reference to a metric scale or “linescale.” The system displays a visual indicator along the scale corresponding to the first existing product rating as represented at S50. This may be repeated for several different aspects of the existing product.

[0071] At S60 of this example, the system displays a “header concept” or general introduction of a new product. While reference is made to a new product, it will be understood that the new product could simply be different from the first product. The system receives from the respondent and displays ratings of the new product on the predetermined scale used for the existing product at S70. As generally illustrated, this is repeated for several different aspects of the new product

[0072] Once the ratings of the new product are displayed, the system preferably displays a detailed description of the new product at S80. The respondent then is prompted to again rate the new product at S90 and a visual indicator of the rating of the new product is displayed at S100. Preferably, this is performed successively for various aspects of the new product.

[0073] Once the ratings are collected, at S110 the system generates a score, preferably by taking the average of the total responses and multiplying them by, for example, 100. The system then displays the score and values to the respondent at S120. Preferably, the respondent is prompted to validate the scores at S130, for example, by indicating whether the respondent agrees or disagrees with the scores. Preferably, the respondent can also provide reasons for the ratings and other comments.

[0074] According to another aspect of the invention, illustrated generally at S140, the system collects and processes the scores from several respondents to generate a report. Aspects of the report may include those described more specifically below in reference to one exemplary embodiment.

[0075] An example of one embodiment of the invention is now described in reference to a bill handling product. As generally indicated above, respondents are first contacted and requested to participate in a survey. For example, the system may generate solicitation notices to potential customers in the target market by electronic mail. For example, the message indicates that the request is for surveys used for future product ideas. Preferably, it includes an incentive for participation, such as entry in a prize drawing. Preferably, the message provides an overview of the survey, for example, indicating that it will: ask the respondent to indicate what he or she does today on a continuous metric scale; show new ideas and ask how to rate them on the same continuous metric scale; recap the rating and what it means and ask the respondent whether he/she agrees or disagrees. Preferably, included in the message is the URL for access to the survey.

[0076] Once the respondent begins the survey, the first screen asks the respondent questions related to his or her current bill payment practices. For example, it may ask: “Are you the person who pays the bills in your household?”; “About how many bills does your household pay per month?”; “How do you usually handle your bills?”; and “You sometimes also . . . ”. As illustrated in FIG. 3, the respondent responds by selecting from answers provided on a drop down menu or in a text box for other responses. In the example shown in FIG. 3, the respondent has responded that he is the bill payer, he pays nine bills a month, and pays most by check, but a few electronically.

[0077] According to one feature of the invention, the system then displays a predetermined scale against which the respondent can indicate a rating for an existing product. Preferably, the scale is based on the information previously furnished by the respondent in the preceding screens (such as illustrated in FIG. 3). A series of such continuous metric scales are shown in FIGS. 4A-4C. For example, FIG. 4A, illustrates a screen which states the respondent's current practice of bills coming in the mail and paying most by check, and prompts the respondent to rate his or her experience of paying bills by check on a metric scale that runs from excellent to terrible. Typically, the respondent may indicate his or her preference using a mouse to select a point on the scale and then confirm the result. While a mouse is described above as a user input device for indicating ratings on a continuous metric scale, it would be apparent to one skilled in the art that any number of other electronic input devices could also be used in the present invention.

[0078] The system then validates what the respondent has said by marking the continuous metric scale with the respondent's response to the question of how the respondent rates his or her current bill payment experience of paying bills by check. In this example, the respondent rated the experience as “poor.” On the same screen, the system asks the respondent to rate his on-line bill payment experience on the same continuous metric scale from excellent to terrible.

[0079] Next, as shown in FIG. 4C, the second rating for paying on-line is displayed simultaneously with the first rating. the system displays to the respondent both of his responses on one continuous metric scale. In this example, the respondent reported that his experience of paying on-line is good and that his experience of paying by check is poor. In addition, the system prompts the respondent to “Click here to see the concept.”

[0080] The system next displays to the respondent a “headline” concept statement about the new product. An exemplary headline concept is as follows:

[0081] Finally, an easy way to take the drudgery out of getting your bills paid on time. With this new service you can get your utility bills, credit card bills, telephone, Internet ISP, Cable, student loan bills, insurance—even mortgage or rent—right on the web! You'll get an email as soon as any bill or invoice is ready for viewing. Then you go to your private web site located behind your bank's security firewall. Since you can set up reminders and alerts to notify you anyway you like (email, instant messenger, even a text message sent your cell phone!), you won't have to worry about missing a payment or a due date ever again. Although you can access all you relevant bills and documents in one easy place, each bill is coming directly from your biller. No one else, including your bank, can see or access the information.

[0082] You simply log on to your personalized web site and you can:

• • [0083] View all the details of your bills and statements
  • [0084] Pay bills electronically right from any bank account or charge them to a credit card
  • [0085] Store these bills and documents online for future reference
  • [0086] Print bills and statements for your paper files
  • [0087] View a complete organized payment history and archive
  • [0088] Share a specific bill or statement with someone else—for example, share the tuition bill with your parents

[0089] This service will present and archive the following bills and statements to make your life a little more manageable.

• • [0090] Utility bills (phone, electric, cable, etc.)
  • [0091] Rent or mortgage
  • [0092] Car Loan bills
  • [0093] Credit Card bills
  • [0094] Insurance bills
  • [0095] Medical bills
  • [0096] Tuition bills
  • [0097] School Housing, Meal Plan, Student Health, Bookstore bills
  • [0098] And practically any other bill from any business or institution

[0099] After the respondent reviews the concept, a next screen (see FIG. 5) prompts the respondent to rate the concept on a continuous metric scale that reflects the respondent's previous rating of his current bill paying experience. The respondent's ratings of his current experience are called “metric anchors.” This innovative feature helps get valid and replicable data from the respondent. By recording the respondent's view of the respondent's current experience, the respondent articulates his or her baseline experience on an objective basis.

[0100] Once the respondent marks his rating, the system displays a screen that reflects the respondent's rating, as shown, for example, in FIG. 6. In this example, the respondent rated the New Concept as very good. On the same screen, the system asks the respondent to “Click here to see how it works.”

[0101] Next, the respondent views a more detailed description of how the new product or concept works as shown in FIGS. 7-11. Preferably, the system describes the product in a detailed narrative, for example, as illustrated in FIG. 7. Preferably, the system subsequently displays a series of interactive screens, which provide tasks to the respondent that give the respondent a typical experience with the new product.

[0102] In this example, interactive screens relate to a bill payment system for a bank. It includes a series of actions that can be performed for a checking account, including a summary of accounts, detail of accounts, account summary, account download, stop payment, order checks, transfer funds, and pay bills. The screens include a drop down menu to select from existing accounts. The available balance is also displayed.

[0103] As shown in FIG. 8, a personalized welcome screen displays a series of new bills, including the biller, the date received and their status. The screen includes option buttons to add a biller, delete a biller, and edit preferences.

[0104] FIG. 9 illustrates a screen for selecting the method of payment. FIG. 10 illustrates a screen for viewing a bill. FIG. 11 is a partial view of an account summary page. It will be understood that the illustrated screens are simply examples of what may be displayed to permit the respondent to experience a new product.

[0105] At the end of the experience, in the example of FIG. 11, the system prompts the respondent to “Click here to continue.”

[0106] On the next screen, as shown in FIG. 12A, the system prompts the respondent, “Now that you know more about the idea, how do you rate it—higher or lower?” The screen prompts the respondent to place the respondent's rating of the more detailed understanding of the concept and the concept experience on the same scale as the respondent's rating of the concept and the respondent's two current favorites.

[0107] Once the respondent inputs a rating, the system, as shown in FIG. 12B, tells the respondent how the respondent rates the new concept overall. In this embodiment, the respondent rates the New Experience as Very Good and better than his rating of the New Concept and his current bill payment practices of paying on-line and paying by mailing a check. In the same screen, the system tells the respondent to “click here” for a few more questions.

[0108] The system then asks the respondent to rate the new idea on a series of continuous metric scales. For example, a series of four lines scales may be provided to allow the respondent to successively indicate: ease of use, speed, charm, and whether the respondent would recommend the new product to friends. It does this by asking the respondent to answer the following questions: “How easy or hard is it to use?”, “How fast or slow is it?”, “Is it charming and pleasant or ugly?”, and “How likely are you to recommend it to friends?” For each answer, the system generates a score and displays the answer and the score to the respondent. The final screen showing all four answers is shown illustrated at FIG. 13. This feature ensures that the respondent understands what his answer is and the score.

[0109] Although a last screen of this sequence is shown in FIG. 13, preferably, the respondent is prompted on a step-by-step basis. For example, first the system asks, “Based on what you know so far about the idea and how it works, how would you rate it for ease of use? Click on the line from ‘Easiest’ to ‘Terribly Hard’ to tell us what you think.” In this example, the respondent rates the idea as Very Easy, and the system generates a score of 41.4. The system then responds with a screen confirming the respondent rates the idea as Very Easy and tells the respondent that the score is 41.4. On this same screen, the system asks the respondent the next question, “Based on what you know so far about the idea and how it works, how would you rate it for speed? Click on the line from ‘Fastest’ to ‘Painfully Slow’ to tell us what you think.” In this example, the respondent rates the speed as “Very Fast,” the system generates a score of 44.4 and displays a confirmation screen depicting the respondent's rating and score. In the same screen, the system may ask the next question, “How pleasant was the idea? Click on the line from ‘Charming’ to ‘Ugly’ to tell us what you think.”

[0110] In this example, the respondent rates how pleasant the idea was as “Very Pleasant.” The system generates a score of 44.7 and displays a confirmation screen, which shows the respondent's rating and score. On this screen, the system asks the final question, “Would you tell your friends about the idea? Click on the line from ‘Yes’ to ‘No way’ to let us know.”

[0111] In this example, the respondent marks it as, “Yes, I would absolutely tell my friends about this and ask them to try it.” The system generates a score of 41.2, and displays a confirmation screen, such as FIG. 13, which shows the respondent's rating and score. It asks the respondent to “Click here to finish.”

[0112] In the preferred embodiment, the system transmits respondent's responses over the network to the host computer. Once information corresponding to the respondent's responses is received by the host computer, it calculates the score by recording a number for each response on the continuous metric scale to questions about the new product concept and more detailed explanation. The system computes the score by taking the average of the total responses and multiplying by 100.

[0113] According to another feature of the invention, the system then provides the respondent a report, such as in the form shown in FIG. 14. In this example, the report indicates to the respondent that the respondent's personal rating for the concept was 750, out of a scale from 200 to 800.

[0114] Preferably, the system tells the respondent what the score means in easy to understand language. In this example, it says:

[0115] “Here are the scores and their values:

[0116]

200 to 300	Forget about it
301 to 400	Not ready for prime time
401 to 500	Something here, but it needs work
501 to 600	Good concept, works pretty well
601 to 700	Very strong idea, great execution
701 to 800	I'm wild for it

[0117] In this example, the system asks the respondent whether he agrees with his score, and whether he would rate it higher or lower. The system then asks the customer for the reasons for his rating to be typed in a text box. Then the respondent is asked to select a number of reasons from a checkbox list of possible reasons for his rating.

[0118] As shown in FIG. 15, the respondent is then prompted to provide certain information to help assess the respondent. As illustrated, these may include demographic and relevant behavior questions, as well as questions to gauge the attitude of the respondent.

[0119] From the foregoing, it will be appreciated that the “Linescale” technique can be used advantageously to evaluate both concepts and products. It will further be appreciated that various ratings can be utilized on the scale, including descriptive ratings as well as numerical ratings. Further, different numerical values or weights may be attached to particular rating points on a scale. Preferably, the ratings are assigned a corresponding score that can be used to provide an overall evaluation. Preferably, the respondent views each continuous metric scale successively so that a respondent can provide ratings for a first, existing product or concept, and then rate a new or different product or concept against the same scale.

[0120] Another aspect of the invention is the detailed and automatic report that the system generates for the product developer or user of the system. This report is referred to herein as a product assessment report. The report preferably can be furnished using a web-based system as illustrated generally in FIG. 1, wherein the host computer 10 of the system transmits the report to the terminal 12 of the product developer or user of the system over the network 11. Preferably, the host computer may also notify the user by email that the report is ready, and the user accesses the report from the host computer over the network. Other implementations for delivery of the report will, of course, be apparent to those skilled in the art.

[0121] Preferably, data is reported automatically in a decision rule-driven reporting format. Preferably, reporting is real time. As each respondent answers, his responses are added to the database, and new calculations are automatically generated. There are both individual concept and product reports, and an aggregate “Scorecard” which displays a number of concepts or products and their scores.

[0122] An example of such a report is shown in FIGS. 16-30. In this report, the system automatically does the following: summarizes the new product concept that was shown to the target customers or other respondents who took the survey (see FIGS. 16 and 17); reports the scores in the correct tables, and the reasons for the score, (see FIGS. 16-30, wherein scores are referred to as “I-CAT scores), and generates table headings and table heading narratives in plain English acknowledging sample size, preferences, percentages, scores, etc., based on decision rules.

[0123] As illustrated, the system interprets the data and analysis and reports this interpretation in plain English, based on decision rules. For example, if the preference for the experience over the concept is higher than a predetermined percentage, the analysis states, “The product delivers on the promise.” Examples of other automatically generated messages, based on predetermined percentages include, by way of example, “The product supports the concept, but needs improvement,” or “The product does not deliver on the promise.”

[0124] The system also creates a unique scorecard generator, which allows for the custom creation of a scorecard with comparisons to scores of any selected concepts or products. It further allows the user to change descriptive names of the selected concepts for confidentiality or other purposes. The system allows or denies linkage to the individual report supporting the score. The system provides the data in rank order determined by the top score. The system highlights key information, such as the Score Segments and reports the size of the market segment among which the test was conducted. The system also allows the user to edit the reports.

[0125] With this report, the product developer has the information needed to decide whether to modify the product, keep it as is, or discard it.

[0126] As noted above, embodiments within the scope of the present invention include program products comprising computer-readable media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, such computer-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above are also to be included within the scope of computer-readable media. Computer-executable instructions comprise, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions.

[0127] The invention is described in the general context of method steps, which may be implemented in one embodiment by a program product including computer-executable instructions, such as program code, executed by computers in networked environments. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.

[0128] The present invention in some embodiments, may be operated in a networked environment using logical connections to one or more remote computers having processors. Logical connections may include a local area network (LAN) and a wide area network (WAN) that are presented here by way of example and not limitation. Such networking environments are commonplace in office-wide or enterprise-wide computer networks, intranets and the Internet. Those skilled in the art will appreciate that such network computing environments will typically encompass many types of computer system configurations, including personal computers, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination of hardwired or wireless links) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0129] An exemplary system for implementing the overall system or portions of the invention might include a general purpose computing device in the form of a conventional computer, including a processing unit, a system memory, and a system bus that couples various system components including the system memory to the processing unit. The system memory may include read only memory (ROM) and random access memory (RAM). The computer may also include a magnetic hard disk drive for reading from and writing to a magnetic hard disk, a magnetic disk drive for reading from or writing to a removable magnetic disk, and an optical disk drive for reading from or writing to removable optical disk such as a CD-ROM or other optical media. The drives and their associated computer-readable media provide nonvolatile storage of computer-executable instructions, data structures, program modules and other data for the computer.

[0130] Software and web implementations of the present invention could be accomplished with standard programming techniques with rule based logic and other logic to accomplish the various database searching steps, correlation steps, comparison steps and decision steps. It should also be noted that the word “component” as used herein and in the claims is intended to encompass implementations using one or more lines of software code, and/or hardware implementations, and/or equipment for receiving manual inputs.

[0131] The foregoing description of embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. The embodiments were chosen and described in order to explain the principals of the invention and its practical application to enable one skilled in the art to utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated.