December 19, 2008 ~ Market researchers commonly use a mathematical technique called intent scale translations to convert a respondent’s stated purchase intentions into actual purchase probabilities. Intent scale translations provide the researcher with an estimate of actual buying behavior and accounts for over estimation on the part of respondents’ participation in the research. Companies can then rely on purchase intentions to forecast the purchase of new products or repeat purchase of existing products. In other words, if a customer states that he/she prefers a particular product over another, what is the probability that he will actually follow through with making the purchase of the preferred product?

Studies have shown that consumer’s self-reported intentions to purchase do not reliably predict their purchasing behavior (1). Market researchers needed to develop a method to more accurately translate a respondent’s response to survey questions into actual probabilities that they would buy or use a product of interest. Intent scale translations take data from a customer survey on purchase intentions and convert the data into a prediction of purchase probability by using comparisons of stated vs. actual purchase behavior.

Traditional intention rating scales use a 5-point scale to show consumer’s intentions of buying a product. In this traditional method, people who scored a rating of 1 or 2 were typically assigned a 0% chance of buying the product. The research of Thomas Juster (2)found that the 5-point scale was an inaccurate way of measuring buyer intention and that a buyer with a score of 1 and 2 actually had a greater than 0% chance of buying the product. Juster created his own buyer intention scale that would improve a marketer’s ability to forecast behavior from intentions and account for changes in a consumer’s true intentions between the time surveyed and the time of actual purchase. Juster developed an intention scale that adjusted the intention scores by analyzing the actual future purchase behavior of consumers after being surveyed. He developed a new 11-point scale which implied simple methods for calculating intentions. It stated that if a sample group was asked if they would buy a new car in the next month and they were to choose a number correlating to the likelihood on a scale of 0 to 10, 10 being the greatest possibility, and the average score came out to be 2.5 out of 10 then that would translate to 25% of the general population purchasing a new car in the next year. The research described here will provide a demonstration of how purchase intention scales are used to predict actual consumer purchase behavior.

The traditional 5-point scale is used to give a general estimate on consumer’s purchase intentions of something. It can be used to predict everything from consumer’s intention on buying a certain product to predicting where people may travel for vacation or even the likelihood of a doctor ordering a particular diagnostic test for a patient. The 5 point scale is routinely used across many types of businesses and industries. Below is a typical scale and description of a 5 point Intention Scale.

Table 1. Example of 5-Point Intention Scale

The 5-point scale, also referred to as the Likert scale, is still commonly used by major corporations to understand consumer’s product purchasing intentions or simply conducting basic surveys. It is especially useful in telephone surveys and mall research where consumers are taking a verbal survey and their responses are being recorded by the researcher.

One major corporation that still uses this 5-point scale is AC Nielsen, a marketing research company that conducts surveys on a variety of consumer products. AC Nielsen provides a service called BASES to present pre-market insights to consumer goods companies. Consumer good manufacturers often use outside market researchers for conducting unbiased surveys. BASES has become the industry standard forecasting model. Table 2 demonstrates how a 5-point scale is translated into purchase probability. Note that Intent Probability overestimates predicted probability when comparing to the AC Nielsen BASES translation scale (3). AC Nielsen and others have conducted diary studies where consumers record their actions for the researcher over time and follow-up market research to measure actual observed purchase behavior compared to stated intentions. This has resulted in establishing a correction factor that adjusts the intent probabilities of purchase.

Table 2 Use of 5-Point Intention Scale in Translating Purchase Probability

As previously mentioned, another common purchase intention scale used is the 11-point scale, created by Thomas Juster, which he found to be much more accurate than the 5-point scale. On the Juster scale, every description correlates directly to a number ranking of 0 to 10. The reason for this alternative scale is to give higher values to people ranking lower scores. In the 5-point scale, someone who receives a score of 1or 2 were sometimes assigned a 0% chance of using the product. Juster found this to be incorrect and set out to fix it by coming up with his own scale. In the 11-point scale in Table 3, the score relates directly to the probability of use. For example, someone with a score of 4 is found to have a 40% chance of using the product.

Table 3. Juster’s 11-Point Probability Scale

Herein we report a review of consumer studies as it relates to a Translated Probability Intent Scale. What I found was a research study on Fast-Moving Consumer Goods. This particular study observed three popular types of soups and four types of yogurt. It involved a face-to-face survey performed by the Palmerston North Household Omnibus survey (4). The consumers were asked about their purchase intent of each of the items and an overall percentage of purchase probability was obtained. Then the respondents were re-interviewed by telephone a month after the original survey and estimates of actual purchase intent were obtained. For example, Fresh and Fruity yogurt had a predicted purchase rate of 36.2%, but when the respondents were re-contacted, only 22.6% had actually purchased the Fresh & Fruity yogurt. This indicates an overestimation of expected purchase intent by13.6%. Table 4 shows the results from this survey for various products.

Table 4. Purchase Intention Using Juster’s 11-point Scale

We plotted all of the actual and predicted probabilities on a scatter plot with Predicted Purchase Intent on the X-axis and Actual Purchase Behavior on the Y-axis. In Figure 1 the graph and equations generated from the 11-point Juster scale results are shown.

Figure 1. Juster 11-Point Scale: Predicted vs. Actual

Each mark represents a unique study. A “best fit” regression line was applied to the resulting plot using the best fit line feature in Excel and the equation of the line was calculated. The regression line shows y = 0.8845x – 0.0481. The linear correlation coefficient, r, gave a strong correlation coefficient of r = 0.9713. A perfect correlation is + 1 where all points lie on a straight line. A correlation coefficient greater than 0.8 is considered strong (Marino 149). The coefficient of determination, R2, was calculated and found to be R2 = 0.9435.

We took the Juster Scale Intent Probability numbers in Table 3 and entered into them into the regression equation, y = 0.8845x – 0.0481 (x= 0.99, 0.9, 0.8, etc). These represent the predicted purchase probabilities “x” in the equation. The decimals are then converted back into percentages for the final Translated Probability Table 5 below:

Table 5. Just Scale Translated Probabilities

From this analysis 83% of actual intended buyers (those with a scale score = 10 described as certain purchasers) can be expected to buy the product vs. number predicted. Of those with a score of 9 and intent probability of 90%, only 75% would actually be predicted to purchase the product. Of those with a score of 5 and intent probability of 50%, only 39% would actually be predicted to purchase the product. This demonstrates how statistics can be used in intention scale translations to better predict a consumer’s actual purchase intent from survey data. Statistics like this are used by market researchers to help guide marketing, advertising and sales forecasts for product manufacturers.

To download a PDF version, click here.

Bibliography

1. Sheppard, Blair H., Jon Hartwick, and Paul R. Warshaw. “The Theory of Reasoned Action: A Meta-analysis of Past Research with Recommendations for Modifications and Future Research.” Journal of Consumer Research 15.3 (1988); 325-343.

2. Juster, F. Thomas. “Consumer Buying Intentions and Purchase Probability: An Experiment in Survey Design.” Journal of the American Statistical Association 61 (1966); 658-696.

3. Chandon, Pierre, Vicki G. Morwitz, and Werner J. Reinartz. “Do Intentions Really Predict Behavior? Self-Generated Validity Effects in Survey Research.” Journal of Marketing 69 (2005); 1-14.

4. Brennan, Mike, and Don Esslemont “The Accuracy of the Juster Scale for Predicting Purchase Rates of Branded, Fast-Moving Consumer Goods.” Marketing Bulletin 5 (1994) : 47-52

5. Marino, Kenneth E., Forecasting Sales and Planning Profits. Chicago: Probus Publishing, 1986.

6. The Nielsen Company. Marketing Intelligence. 13 Dec. 2008 <http://www2.acnielsen.com/site/index.shtml>.

Abstract

The adequacy of labeling for instructions, cautions, and contraindications becomes pivotal in the benefit/risk assessment of medical products. Our research using observational study designs in mock product use trials has predicted medication use error rates in the range of 40-70% for drug device combination products. Early testing of proposed labeling can identify problem areas in label comprehension that leads to a potential medication or administration error or user frustration resulting in medication non-compliance. A well designed labeling comprehension study is not an expensive proposition; most studies can be performed with a small sample size (n < 100) for baseline observations and testing of label revision(s). In our experience, such studies support labeling changes that can significantly reduce potential product misuse and medication error to acceptable rates.

1. Introduction

Initiatives to reduce health care spending have ultimately resulted in many medical procedures increasingly being moved to the outpatient environment and/or home care setting. As such, patients, family members and/or other non-medically trained personnel are performing therapeutic and diagnostic procedures such as drug administration using alternative drug delivery devices, inhaled therapies, glucose tests, etc. Adequacy of the labeling instructions, cautions, and contraindications and the use of supplemental instructional labeling have become pivotal in the benefit-to-risk assessment of these treatments or tests. This paper describes a research methodology offered by BioTrak Market Intelligence, Inc. (BioTrak) for evaluating the effectiveness of combination drug/device product labeling prior to product registration as an aid in the development of appropriate labeling and during the post-approval period to assess comprehension and compliance among patients, dispensing pharmacists and patient care providers.

The medication errors staff in the FDA Office of Post-Marketing Drug Risk Assessment (OPDRA) search the FDA Adverse Event Reporting System (AERS) database for all cases of medication error. In a report published in 2001 (Drug Topics October 2001, p 23-24), labeling was identified as one of the leading causes (20%) of medication error along with misinterpretation of the order (10%) and written communication (8%). Additionally, human factors are the leading cause (42%) of comprehension and performance deficit. These data illustrate the importance and need for clear use labeling.

2. Regulatory Context

The FDA has promulgated detailed regulations specifying the form, content and wording of labeling for items such as the identity, dosing, supporting studies, warnings, adverse reactions, contraindications with respect to established drugs and biologics dispensed by a pharmacy or sold over-the-counter (OTC). (Title 21 Subchapter C-Drugs: General Part 201 Labeling). Likewise the labeling requirements for medical devices are specified in Part 801 Labeling. The FDA has required sponsors to thoroughly investigate the adequacy of labeling for OTC switches of prescribed medications. For example, FDA and consumer groups required sponsors to conduct five labeling comprehension studies and five actual use studies before the approval of Prilosec® for OTC marketing. FDA plans to conduct labeling comprehension studies of their own prior to issuing new regulations for covering the wording of statements that request patients to report adverse events. Among the reasons cited for testing these statements were: (1) to determine the best and most precise wording for the statements, (2) to evaluate consumer comprehension of the proposed statements, and (3) to address concerns that consumers who read the statement will mistakenly call FDA in search of medical advice (Federal Register / Vol. 72, No. 22 / Friday, February 2, 2007 / Notices).

For combination products sponsors are expected to submit adequately well controlled scientific evidence for the safety and effectiveness of their labeling. However, to date the FDA has not issued guidance for combination product labeling comprehension studies.

3. Strategies for Labeling Studies

In today’s climate of increasing regulatory control, authorities are demanding more controlled studies to evaluate and verify product safety and the accuracy of product use instructions by all product stakeholders involved in the prescribing, dispensing and administration of the product. Common methods used to evaluate labeling comprehension (prescribing, dispensing and use instructions) typically include one or a combination of the following:

Product Use Simulation Studies – Proposed labeling is drafted and applied to test articles for hands-on evaluation by target stakeholders required to take specific action(s) demonstrating proper administration of the proposed product (drug, device or drug/device combination product).

Your browser may not support display of this image.

Ailment Panel Survey Research – Patients (or patient caregivers) who are potential users of the proposed drug, device or drug/device combination product are surveyed (online, paper-based MD office exit interviews, etc) regarding their interpretation of specific use instructions using both objective and subjective queries.

Patient Registries and Post-Market Surveillance Studies – Once the approved drug, device or drug/device combination product is commercialized, sponsors are frequently required to monitor compliance for a period of time (1 to 3 years is common) to generate patient experiential data. Depending on the risk level of a labeling compliance error, post-market surveillance studies can either be mandatory for all product dispensed (high risk for severe AE), or, in the case of a relatively low risk potential, the post-marketing surveillance can be self-reported by the patient or caregiver on a voluntary participation basis.

4. Case Studies of Labeling Comprehension Trials

Studies of labeling comprehension, compliance and product usability have revealed surprisingly poor product safety and performance results by patients, caregivers, and pharmacists. Figure 1 below is an example of a caregiver study involving manipulations with a novel drug delivery device. One device represented the currently marketed product; the other four devices tested were replacement prototypes. The study revealed labeling as a major source of end user confusion.

Several case study examples follow which we have found indicative of user performance with initial labeling and new product designs. Each example involves a drug with novel delivery device for administration by a patient or caregiver.

Case 1: Pharmacist Study of Dispensing Error with an Adjustable Drug Delivery Device

A pharmaceutical company had developed an adjustable dosage delivery device for an approved medication to treat an acute disorder. The adjustable device was developed to reduce the range of inventory SKUs while offering more available dosing increments. Prior to use, the device required pharmacy setting and locking of the patient specific dose. The device was made available as a pack of two set at an arbitrary default dosage; pharmacists were to set the device at each prescribed dose and lock it. Labeling consisted of an outer box label and insert providing a diagram accompanied by a step by step written procedure. The principle failure modes were identified as 1) failure to set the correct dose resulting in incorrect dosing and 2) failure to lock the device thereby disabling its use.

Two studies were conducted to simulate as close as possible real practice dynamics to evaluate the labeling effectiveness. In each study, a mock prescription order and final packaged product was handed to a pharmacist in their own setting and they were given a reasonable period of time to “fill the prescription”. The study monitor collected the filled prescription, scored the results as correct or incorrect, and took observations of pharmacist behavior while setting and locking the device. An initial study of one hundred and one retail pharmacists was performed to test labeling comprehension.

As shown in Figure 2, the initial study (Study 1) demonstrated a 42% rate of dispensing error, signaling a need for labeling revisions to the outer box and instructional labeling. Following label revisions and beta testing of the revised articles as part of the design control process, a second study (Study 2) was performed with fifty retail pharmacists to measure effect of the labeling changes with dispensing pharmacists. The comparative results for study 1 and 2 are shown in Figure 2. The error rate from study 1 to study 2 declined from 42% to 8% based on the effect of labeling changes.

Case 2: Patient Self Administration with a Novel Drug Delivery Device

A pharmaceutical company had developed an adjustable dosage delivery device for an approved medication to treat a common disease. The device was developed to offer patients more convenience and portability of dose administration and potentially more accurate dosing. Prior to use, the device required priming and setting of the patient specific dose by the patient or caregiver. The device was initially set at an arbitrary default dosage.

Three product use trials were conducted with pre-registration materials. An initial study with forty patients was performed to test labeling comprehension and product usability with the target patient population. This study (blue bars in Figure 3) identified several issues with label comprehension with just 31% performing device setup correctly and only 67% correctly administering doses from the device. As a result, labeling revisions were made to the device and instructional insert, and a second study was conducted with forty-one patients to retest label comprehension and product usability. This study (red bars in Figure 3) demonstrated improvement, however the results remained unsatisfactory. More significant labeling and instructional insert changes were made by the pharma company, and a third study with 20 subjects was conducted to confirm the improved efficacy of the labeling revisions. The comparative results for studies 1, 2 and 3 are given in Figure 3. A dramatic improvement was observed from study 1 to study 3 with overall error reduced in half or more for each key measure, including a 95% success rate with dose administration in the last study.

5. Discussion

BioTrak has conducted other studies similar to those reported here with initial study observational product use error rates typically in the in the range of 40-70% for respondents naive to a novel drug delivery device. Sponsors are often shocked at these findings which suggest far greater label comprehension issues than what can be estimated from calls and complaints to medical affairs departments. A well designed labeling comprehension study is not an expensive proposition, most study designs can evaluate labeling materials with 40-100 subjects before and 40-100 subjects after label revision. These studies demonstrate the benefit of conducting label comprehension studies as a means for improving product safety, efficacy, and usability.

6. Conclusions

A well designed mock trial where product use is simulated has in our experience correlated well with actual field results. Such studies can provide a significant reduction in product use errors with the potential outcome of improved safety, efficacy, and competitiveness. Early testing of proposed labeling can identify problem areas in label comprehension that lead to a potential medication or administration error and/or frustration with product use. Such studies validate design control, quality of labeling and can facilitate the product registration process.

For more information contact BioTrak at 760 448-4823

[View PDF]