What Is a Legitimate Review?

A version of this discussion has come up several times in posts and comments on this blog: What is a legitimate review? Miriam Ellis offers a thought provoking post in which she suggests that a site like Yelp might remove multiple positive reviews coming from a single computer or IP address as suspicious.

The scenario she paints is this: a business owner offers a “public” computer on premises or even free WiFi and enourages people to write reviews, directly or indirectly. But the review site, in this case Yelp, might see a bunch of positive reviews coming from a single IP address and regard them as illegitimate.

Clearly if I as a small business owner pretend to be another person and write a fake review lauding my own business or if I get friends and family who are not customers to do something similar that’s “verboten.”

But what about a situation in which I provide an incentive of some kind to write a review while in my shop/shore? How about a “free cup of coffee to anyone who writes a review on Yelp?” This type of incentive was being used — as in free Starbucks cards — by a number of the early review sites (InsiderPages, Tribe, etc.) to get people to write reviews.

In my hypothetical no one is being asked to write a “positive” review, just to write “a review.” Say my program is extremely successful and suddenly (the week I start it) 30 or 40 mostly positive reviews show up. How would Yelp or another, similar site respond? If the reviews are written on-site using free WiFi Miriam’s post suggests they might raise suspicion coming from the same IP address and be removed. But should they be removed because of the incentive alone or would they if they appeared en masse in a relatively compressed time frame?

Let’s put aside the IP address issue for a second. How would my incentive program be treated? In my example these are all real people. I as the business owner haven’t asked them to say anything in particular, especially anything positive. I might even go out of my way to say “tell us what you really think, good or bad.” Yelp does explicitly address this situation and doesn’t like it either way. It asks users to disclose incentives and discourages them from writing reviews in response to such a program:

What if the business I’m reviewing gave me something for free or at a discount?

You should never accept freebies or discounts in exchange for reviews. For example, if a bar owner offers you a free drink in exchange for a 5-star review, you should not accept his or her offer.

Of course it’s ok if you were given something for free or at a discount independent of your review, but you should always disclose any special treatment, gifts, or discounts in your reivew. For example, if the restaurant manager gave you free appetizers on opening night, you should include that information in your review. Yelping is about real, honest reviews, so while we’re happy you had an extra special experience, we ask you to tell us (your readers) the whole story.

I’m in the midst of another survey of small business advertiser attitudes. To the question (also asked last year), “What do you think of online reviews?” the top response so far is “online reviews are a good thing and are helping us improve our business.” That was the winner last year as well:

Attitude of SMBs toward user reviews

As you can see we’re in a gray area ethically and philosophically with some of this behavior. But the stakes are high for local businesses, as consumers increasingly look at reviews as part of their decision-making process. In a city like San Francisco, Yelp arguably has more influence than any other single site.

What is needed is additional clarity regarding what’s permissible and what’s not (e.g., “reviews coming from the same computer will potentially be disallowed”). Most businesses will be ethical — just like most people are ethical — but they’re trying to figure all this out. I certainly don’t blame them for being creative.

Advertisements

30 Responses to “What Is a Legitimate Review?”

  1. Mike Orren Says:

    This is an issue that comes up a lot for us — And we don’t remove any “fake” reviews, but add notes to help users better decide.

    For instance, if we get a bunch of reviews from the same IP we post a comment like:

    “We haven’t personally visited this restaurant, so can’t speak to the content of the last two reviews, but we do know that they came very close together, from new users, and from the same IP address. That means it was likely, but not certain, they were posted from the same computer.”

    And we try to educate our restaurant owners on how to deal with user reviews — Hell, we’ve practically given a tutorial on how to fake it and not get caught. But the beat goes on…

    When all else fails, we send ’em this video:

    http://www.pegasusnews.com/blogs/pegasusnewsblog/2008/apr/18/fake/

    I should also note that as the community grows, there’s a fair amount of self-policing. Although our users don’t have access to the information we have to bust fake reviewers (ip, email address), they’re pretty savvy about noting reviews that seem off. That means that the review stays up, but there’s extra info.

    One final note: The easiest way to tell if a review is fake is its content. Fakes are almost always written from the POV of things a restaurant owner wants you to care about but that a customer would never notice. For example:

    – Names of servers, managers and non-celebrity chefs dropped in the interview
    – Extra information like hours and specials, particularly if those mentioned go beyond the range of a specific day.
    – If it sounds like a description from a menu, it probably is.

  2. Mike Orren Says:

    Ack. Don’t know where I got interview from. Should read:

    – Names of servers, managers and non-celebrity chefs dropped in the review

  3. Jeremy, Yelp CEO Says:

    The case of “the computer in the store” is often a cover for… “I created a bunch of fake accounts and wrote reviews and now I’m trying to explain it away.”

    But let’s pretend for a moment that the biz owner does offer a computer in store for reviewing. What does that entail? Well since most people don’t have Yelp accounts, the potential reviewer has to sign up (and give up personal information that they may not be comfortable sharing). Next the user has to confirm their email address. So they’ll need to spend a few minutes digging through their Gmail account on a strange persons computer to find the link. Finally, they’ll be writing a review with the biz owner breathing down their neck.

    So… given the complexity of getting a new user up and running this ever actually happening in a store is a pretty unlikely scenario, but even if this was happening the reviews would be biased in a way that hurts consumers, so we’d do everything we can to stop it.

  4. Greg Sterling Says:

    Yes, over time the volume of reviews should minimize any distortion caused by illegitimate reviews.

  5. Greg Sterling Says:

    Agree with the “public” computer scenario you describe Jeremy. How do you guys treat reviews coming from a single IP (e.g., in store WiFi such as in a coffee house)?

  6. Malcolm Lewis Says:

    Most users reading reviews know to take them with a pinch of salt. Showing the reviewers other recommendations can help increase users’ confidence in any one recommendation.

  7. MiriamEllis Says:

    Hi Greg –
    You’ve really brought this down to the important question of legitimacy. That should be the heart of this discussion, I agree.

    I, too, would be extremely interested to know what Jeremy, Yelp CEO thinks about the wifi scenario.

    In one of the hotels where my family has frequently stayed, there are 2 computers in the lobby for visitor use. Michael Jensen’s suggestion that the computers be set to default to a review page struck me as so smart, and for the life of me, I can’t see anything pushy or wrong in doing this. Google, Yelp, Yahoo, Trip Advisor, etc. have made reviews a prized article, and business owners are going to be trying to think of ways to make it as easy as possible for their customers to leave those reviews. It’s really a usability issue.

    If algorithms aren’t taking into account scenarios such as you, Michael and I are describing, they aren’t meeting all of the vagaries of life.

    I so hope Jeremy will come back on this.

    Thank you for the wonderful post, Greg!
    Miriam

  8. Will Scott Says:

    I wonder if every tactic Yelp has used to increase its rankings in Google would be seen as “legitimate” in the eyes of Google.

    It’s a specious argument in the best case. Users are disproportionately incentivized in the real world.

    The Yelp and Google offense is to treat this new currency as if it has no value — as if it’s egalitarian.

    How do we define ranking earned?
    How do we define reviews earned?
    How do we define public office earned?

    The thought that online we can have some kind of egalitarian utopia is the most arrogant position imaginable.

    If you take people and put them together in a community, online or off, they’re going to work to seek their highest advantage.

    Those of us who do this for a living will presume that those with the greatest number of reviews or highest rankings or most votes will be those who can manipulate the system while appearing legitimate.

    Or, as I’ve said before “In business the most important thing is sincerity, if you can fake that you’ve got it made”.

    Cheers,
    Will

  9. MiriamEllis Says:

    It’s so reminiscent, isn’t it Will, of Google’s stance on paid links? Willfully ignoring the reality that advertising of any kind is nearly always paid, whether it’s a link or a TV commercial. There was an article floating around last week about SEO firms paying to be slated as the top SEO firm or some such thing.

    Is it morally wrong to pay for a review? Is it illegal? Is it nice?

    What is the real question that should decide all of this? I don’t know.

    But, I confess, the idealist in me doesn’t like the manipulation aspect of any of this, even if I’m cynical enough to feel people ought to realize money is at the root of most what one reads and sees. Given the option, I’d like to live in a world where exceptional products and services win real praise and help you, me and the inventor or provider to earn a living.

    I can dream…

    Miriam

  10. Will Scott Says:

    Yes Miriam, it is reminiscent of the paid links hullabaloo.

    And, in an epiphany as I write I’ve come to why:

    Google and Yelp (and others like them) sell ads. In Yelp’s case, the sale of an ad pushes down the natural listings (and may push down bad reviews).

    So, any external attempts to unnaturally affect ranking isn’t just a user experience issue…

    … it’s a profit issue!

    Glass houses, don’t you know.

    Will

  11. Thomas Lund Says:

    I would love to see some statistics on how many of the reviews are bad, average or good. I’ve never been to the US and don’t know how it is there, but here in Scandinavia I’m willing to bet money on that the majority is bad reviews. You remember bad experiences as a customer, but sadly you take good experiences for granted. It might be a culture thing, though.

    When it comes to incentives for giving a review I want to ask you a question. If you have a average experience at a shop, but then get offered a free cup of coffee for giving review, would your experience then go from average to above average?

  12. Greg Sterling Says:

    Most reviews actually turn out to be positive. See these posts:

    http://searchengineland.com/070112-100518.php

    https://gesterling.wordpress.com/2007/06/14/yelp-by-the-numbers/

    but contradicting those is:

    https://gesterling.wordpress.com/2007/12/03/more-evidence-of-reviews-importance/

  13. Todd Says:

    This and the other links are fascinating reading. Two things come to mind.

    1. I’m leary of Amazon’s review totals. When everything is a 4 or 5, they tend to lose their meaning. I just learned you can linger your mouse over the stars to see the actual score. So a 3.8 will show as 4 stars. I usually read the lowest ratings. I also like it when the reviewer (on any site) recommends other similar products.

    2. Are there any studies on the effectiveness of reviews on B2B? I don’t see, very often, customer reviews in our business (wholesaling).

  14. Greg Sterling Says:

    Re #2, probably. I don’t know of any offhand however.

  15. John Says:

    eh ??

  16. Frank, theSUGGESTR.com Says:

    I think a lot of the issue is that without an intrinsic incentive for users to post accurate reviews they will be swayed one way or another towards whatever incentive there is.

    People generally act in a predictable fashion; that is they follow the carrot on the stick. Unfortunately most sites the carrots encourage the use of the site or even worse encourage you to “stick out” of the crowd.

    Obviously as the owner of my own site I have my own vision of how this all should work, but putting that aside I think some of the ‘big’ sites would benefit by making it easy to filter out “suspect” reviews. The term suspect could be defined a number of ways, one of which would be by using a probation period or even better normalizing a users reviews.

  17. Geronimo Says:

    Miriam gives us a great example of a business with computers on the premises to allow legitimate customers to write reviews without any undue pressure from the business owner to write a positive review.

    But that doesn’t mean it’s a good idea for sites like Yelp to allow it if, in doing so, they end up allowing a whole bunch of other unscrupulous business owners to take advantage of the situation and write fake reviews. I’d wager that for every example of a legitimate scenario like the one Miriam paints above, there are ten examples of someone trying to game the system.

    From the consumer’s perspective, I’d gladly sacrifice a business’s ability to allow legitimate customers to write on-premises reviews in order to keep out the many hundreds of sham reviews. After all, the same legitimate customers that Miriam is understandably fighting for can still go home, log on to their accounts, and write a review!

  18. zippy Says:

    Fake reviews are all over Yelp, with more and more being posted every day. Today’s 20-something business promoters regard the posting of fake reviews on Yelp and other review sites as basic marketing. Meanwhile, Yelp has reached the limits of detecting fake reviews from real.

    This makes me wonder about Yelp’s viability. If younger consumers aren’t necessarily believing/trusting the top listed advertisers’ reviews, that means the advertiser isn’t getting much bang for the buck, and therefore may not renew. (While some sponsored ad categories do well on yelp, many others do not.) Plus, why would you, a business owner, pay Yelp to “sponsor” when right after your company’s paid listing, there are 50 other, identical, non-paying competitors with glowing, possibly fake, reviews? And now one of your competitors has just posted a fake, BAD review of your business, which Yelp cannot detect and/ or refuses to remove on the grounds that numerous positive reviews outweigh one bad?
    It’s a lousy sales proposition, and I’m glad I’m not selling Yelp advertising!

  19. zippy Says:

    With regard to the preponderence of “positive” reviews, it’s a cultural thing. From our customer opinion surveys, we find that Americans ALWAYS feel that they must say nice things about people and businesses, even when they don’t mean it and are actually unhappy! In Europe, people are not so relentlessly upbeat– if anything, they have to be pushed to admit that they did, in fact, have a semi-positive experience during their last orgasm. A European Yelp would be filled with people’s contractor horror stories and squeals of pain–for which the name “Yelp” would be much better suited.

  20. Thomas Lund Says:

    Zippy, I agree. At least here in Norway I think that people take above average experiences for granted. Personally, I only remember bad experiences and not the good ones. Which means that I most likely only would bother to write a review if I was very disappointed and felt it was worth the effort.

    Cultural thing, I guess.

  21. Greg Sterling Says:

    RE EU vs. US . . . there are plenty of critical and “snarky” reviews on yelp in my anecdotal experience.

    Re Fake reviews: Zippy, show us what you believe to be a “fake” review and how you’re able to detect it as such. I’m interested to see.

  22. Mihmorandum | 5 Steps for 5 Stars: Reputation Management for Small Businesses | Local SEO Says:

    […] Most Local portals don’t have a clear review policy, but if they do, make sure you know what it is before you engage in a tactic that could lead to a penalty or ban. Yelp specifically does not like incentives of any kind being used for reviews (Greg Sterling, Screenwerk). […]

  23. zippy Says:

    The fake reviews are the ones that read like “I just used Joe Blow Auto Repair, and I could not be happier. The owner, Mr Joseph Blow, is a Platinum Certified master mechanic who treats your car the way that you would like to be treated. They have frequent discount specials (available at wwwjoeblowauto.com) whch mean you can get every day savings on auto repair all year round. All new clients receive a complimentary car wash/free oil change/ride to bus stop.
    I have sent may friends and relatives there, and they all agree that Joe Blow is the best car mechanic in the Bay Area. They are very conveniently located under the freeway. (Take exit, right at taco truck and proceed to end of dirt road.) Thank you, Joe Blow Auto Repair, for keeping me and my family safe and sound at such a low cost. Great experience! http://www.joeblowauto.com.”

    Fake slam

    ONE STAR
    Based on the other reviews here, I took my car to Joe Blow Auto Repair. I have to say that while the place is clean and their pit bulls friendly, they are VERY expensive. They wanted $1,826.00 to change my oi! I took it to two other mechanics who quoted less than half, and ended up paying $431.00 to zippy’s auto repair (they’re great by the way) next to Starbucks on B street.

    :
    Zippy’s Auto Repair: FIVE STARS
    “I’m a former NASA rocket scientist and I appreciate precision, performance and professionalism. We could use the zippys folks down here in Houston. They’ve been working on my 72 BMW 635 CSI and my 74 280z for approximately six and a half years. Terrific mechanics, great people.”

    FIVE STARS
    “Zippy’s really cute, he makes great double decaf lattes and he doesn’t have any icky girly posters on his walls. He is totally honest – and he doesn’t treat you like a bimbo. I just wish he did mani-pedi as well.” zippysautorepair.com

    One obvious way to spot the fake reviews is when the “reviewer” has only ever posted the one review. But even that’s easy to get around. You just write a couple of genuine reviews of places you really have patronized, and now you are a legitimate Yelper. As long as you don’t make any obviously self-serving comments, Yelp cannot distinguish your real reviews from fake ones. This is what today’s 25-year-old car mechanics (plumbers, chiropractors and realtors) are doing to promote their businesses. They create two or three Yelp IDs, enlist a couple of friends in the effort and review themselves to the skies. If they put up 10 positive reviews, the one real, negative review looks like the exception.
    that a good fake review is fake.

  24. CBS Affiliate Hammers Yelp — Again « Screenwerk Says:

    […] and other, similar sites is clarity and guidance about the rules (see this post on what is a “legitimate review“). There’s obvious confusion in the market about what SMBs should expect and […]

  25. Joydeep Says:

    Hi All,

    I found the posts listed here very interesting and insightful. Now I understand that businesses faking reviews can be a major hurdle in creating an honest review website, and identifying these fake reviews may pose as a different challenge for any review website.

    However, I have a question that is not directly related to this discussion. What do you feel about sites that offer points or awards to write reviews, invite friends, etc. Do you think that these sites are legitimate source of information? Do users trust these sites? What challenges would such businesses face in creating an honest site? Do you think review sites should offer rewards to increase traffic? What are pros and cons of this strategy?

    I would really appreciate if you could reply to my questions.

    Joydeep

  26. Managing and Improving Your Business’s Reputation Online - Infoseek Technologies Says:

    […] what it is before you engage in a tactic that could lead to a penalty or ban. Yelp specifically does not like incentives of any kind being used for […]

  27. Lacie Media » Managing and Improving Your Business’s Reputation Online Says:

    […] what it is before you engage in a tactic that could lead to a penalty or ban. Yelp specifically does not like incentives of any kind being used for […]

  28. Anti Aging Products Says:

    oil change needs to be done as often as possible to maintain the good performance of any kind of machinery :`:

  29. Managing and Improving Your Business’s Reputation Online | Targeted Local Says:

    […] what it is before you engage in a tactic that could lead to a penalty or ban. Yelp specifically does not like incentives of any kind being used for […]

  30. 5 Steps for 5 Stars: Reputation Management for Small Businesses | MeetingroomReviewResearch Says:

    […] Most Local portals don’t have a clear review policy, but if they do, make sure you know what it is before you engage in a tactic that could lead to a penalty or ban. Yelp specifically does not like incentives of any kind being used for reviews (Greg Sterling, Screenwerk). […]

Comments are closed.


%d bloggers like this: