The Surprising Power of Online Reviews

His was the first generation to routinely incorporate the online world into the physical one. “Whenever we consider a purchase or going out to a restaurant or doing something new, our first instinct is, let’s read the reviews,” he says.

Likewise, as an academic, Vana is interested in how that online behavior affects what people actually do. In a new working paper written with Anja Lambrecht of the London Business School, Vana shines a spotlight on those ubiquitous online reviews—more specifically, individual reviews—and examines how much they impact consumers’ purchase decisions.

Online reviews have been the subject of much study in the marketing literature over the years. But most of that research has focused on the average ratings of products—say, a blender that received an average of 4.3 stars on Amazon. Vana’s paper represents a new contribution to the field by being the first to study the connection between individual reviews and purchase likelihood.

There are a couple of challenges with doing this type of research. First, Vana and Lambrecht needed good data from a real retailer. This is usually hard to obtain because retailers are loathe to share their customer data with anyone. In this case, the researchers got a lucky break from the Wharton Customer Analytics Institute, which partners with retailers looking for marketing insights. An online retail firm in the United Kingdom (through a third-party data specialist) provided Wharton with its anonymized consumer shopping and purchase data, and Vana and Lambrecht won a grant to use the data in their study. Importantly, the data included click-stream information, which provides a record of when consumers logged on, what they saw during their session, and whether or not they made a purchase. The researchers could tell if consumers read individual reviews of products because the retailer used a pixel tracker that shows how far the consumer scrolled down the product page.

The second challenge is one of endogeneity: when data is the result of non-random activity, it can be a biased representation of behavior. For example, if Vana and Lambrecht studied clickstream behavior from a retailer that used an algorithm to decide the order in which the reviews appear, the outcome would be, in some sense, pre-ordained by the retailer’s motivation to sell the product. Here, the U.K. retailer presented reviews in simple order of recency: the newest review was always displayed at the top. “So, when a new review is added, what was in position 1 goes to position 2, and so on,” Vana says. “We were able to use this change in position of the review to isolate its effect, and we could track the effectiveness of a single review as it changes position over time.”

With those research challenges out of the way, Vana and Lambrecht studied data from the Technology and Home and Garden sections of the retailer’s website, consisting of 380,450 impressions of more than 43,000 reviews during a two-month period in 2015. They found that individual reviews have a huge effect on consumer purchase decisions, with the most power going to the review at the top of the list. For example, they found that replacing a 1-star review in the top position with a 5-star review increases purchase probability by 0.88 percentage points. “This effect is so big, it’s often quite comparable to the effect of aggregate reviews and to running several types of online promotions, such as email coupons, which cost the retailer money,” Vana says.

Within this finding, Vana and Lambrecht make two further distinctions. First, they find that reviews matter most when there is limited information available on the product page. Second, they find that reviews are the most valuable to consumers when they contrast with information consumers might otherwise find on the website.

These findings might lead retailers to think they should just keep 5-star reviews at the top of the list. But Vana cautions against this. When regular customers see great reviews on top all the time, he speculates, they will probably begin to discount the validity of reviews. “Individual reviews have a lot of power,” Vana says. “It’s critical for retailers to think about how to display them in a way that helps consumers find the right product.”