Summary

Ratings help consumers make better choices by learning from the experiences of other consumers. As such, the informativeness of ratings is an important consideration for platform design. However, over time, ratings may inflate and become less informative. Imagine a platform in which a majority of listings have identical ratings. Such a rating system will be ineffective in helping users choose between listings. In this project, we examine the consequences of rating inflation on user purchases, trial, and the concentration of sales in the context of a food delivery platform. We conduct a quasi-experiment that induced rating inflation in a geographical neighborhood and we compare the trends of purchases, trials, and sales concentration in that neighborhood with other control neighborhoods. We find that while rating inflation increases purchases, it decreases trial among users. It also leads to a concentration of sales towards popular restaurants. These findings have implications for platforms that want to increase customer lifetime value by encouraging trial and limiting excess market concentration on their platforms.

Data

Table 1: Data Characteristics
Description N
Number of users 198044
Number of restaurants 2244
Number of transactions 1510739
Number of full pre-experiment weeks 9
Number of full post-experiment weeks 4

Table 2: Summary Statistics (per week)
Variable Description Mean St.Dev. Min Max
n_purchase Purchases by user 0.64 1.23 0.0 29.0
n_trial New restaurants tried by user 0.92 1.00 0.0 14.0
sales Number of transactions per restaurant 67.28 99.87 0.0 1777.0
pre_rating Average restaurant rating pre-experiment 3.53 0.28 2.1 4.4

Illustrative diagram of the delivery platform’s mobile app

Figure 1: Illustrative diagram of the delivery platform's mobile app. The restaurant rating is displayed in the bottom center of each listing. To the left of the rating is the price range of the restaurant, and to its right is the estimated time to delviery based on the user's location.

Figure 1: Illustrative diagram of the delivery platform’s mobile app. The restaurant rating is displayed in the bottom center of each listing. To the left of the rating is the price range of the restaurant, and to its right is the estimated time to delviery based on the user’s location.


Timeline for rating changes

Figure 2: Timeline for rating changes. Original inflated ratings were changed to deflated ratings 14 months before the observation period. First week's data is dropped for being incomplete. Ratings were inflated for the treated region in week 11 of the observation period. During the experiment, inflated ratings were displayed for the Treated neighborhood while deflated ratings continued to be displayed for the Control neighborhoods.

Figure 2: Timeline for rating changes. Original inflated ratings were changed to deflated ratings 14 months before the observation period. First week’s data is dropped for being incomplete. Ratings were inflated for the treated region in week 11 of the observation period. During the experiment, inflated ratings were displayed for the Treated neighborhood while deflated ratings continued to be displayed for the Control neighborhoods.


Rating changes due to quasi-experiment

Figure 3: Rating distribution before and after the experiment for Treated and Control groups. Treated group experiences rating inflation while the Control group does not

Figure 3: Rating distribution before and after the experiment for Treated and Control groups. Treated group experiences rating inflation while the Control group does not


Results

User Activity Levels
(a) Purchases

(b) Trials
Figure 4: User Activity Levels. Top row shows the proportion of users who make no purchase, 1 purchase, 2 purchases, or 3 or more purchases each week. Bottom row shows the proportion of users who have no trial, 1 trial, or 2 or more trials each week. The experiment occurs in week 11.

Figure 4: User Activity Levels. Top row shows the proportion of users who make no purchase, 1 purchase, 2 purchases, or 3 or more purchases each week. Bottom row shows the proportion of users who have no trial, 1 trial, or 2 or more trials each week. The experiment occurs in week 11.


Figure 7: Average Weekly Purchases by Group

Figure 7: Average Weekly Purchases by Group


Table 4: Purchases Increase due to Rating Inflation


Figure 8: Average Weekly Trial by Group

Figure 8: Average Weekly Trial by Group


Table 5: Trial Decreases due to Rating Inflation