How User Reviews Enter a Score Without Hijacking It

Four cleansers, one star rating, thirty points of difference

Four cleansers, all rated by real people, all sitting within a couple of tenths of a star of each other.

↗Dove's Sensitive Skin Beauty Bar carries an enormous review base — the largest crowd in this group by a wide margin. ↗Garnier's SkinActive Micellar Water holds a nearly identical average across a comparably vast pile. ↗Farmacy's Green Clean cleansing balm has the same rating from tens of thousands of people. And ↗The INKEY List's Glycerin Gentle Purifying Cleanser has a marginally lower average from a few hundred.

Ranked by stars and crowd size, that list has an obvious winner and an obvious also-ran. Ranked by our score it inverts almost perfectly: the INKEY List cleanser comes top, Dove close behind, then Garnier, and the Farmacy balm last by a distance.

Roughly thirty points separate products whose public ratings differ by a rounding error. The product with a few hundred reviews outscores the one with tens of thousands, comfortably.

That is not the system ignoring what people said. Every one of those ratings is in the calculation. It is the system refusing to let a star average carry more weight than a star average has earned — and the reason it refuses is that we know quite precisely how star averages are built.

Two biases are baked into every public average

Online ratings are not a survey of everyone who used a product. They are a survey of the people who chose to buy it and then chose to write about it, and both of those choices skew the result in a documented direction.

The first is acquisition bias: people who buy a product are disproportionately people already inclined to like it. Nobody purchases a rich balm cleanser they expect to hate. The population being sampled is pre-filtered for enthusiasm before a single word is typed.

The second is underreporting bias: among people who did buy it, the ones with extreme experiences — delight or fury — are far more likely to write than the large middle group who found it perfectly fine. Research in MIS Quarterly laid both mechanisms out and showed the characteristic result: the familiar J-shaped distribution, a pile of five stars, a smaller pile of ones, and a hollow middle where most actual experience lives. The same work found that consumers do sense the distortion and try to correct for it, but cannot fully do so.

Then there is the money. A 2023 Management Science study of disclosed incentivized reviews — where the reviewer received the product free and said so — found they ran about half a star higher than verified organic purchases of the same items. The disclosure, which is the safeguard regulators lean on, did not neutralise the effect. Worse, the inflation was systematically larger on lower-quality products, which is precisely where a misleading rating does the most damage.

None of this makes reviews worthless. It makes a raw star average the wrong number to trust unmodified.

What happens to a star before it counts

Reviews reach the score through Factor 3 — real-world results — which carries 20% of the overall. Inside that factor, the star signal is 35%, and it arrives only after a sequence of fixed adjustments that apply identically to every product in the catalog.

Verified purchases count at full weight; unverified ones at half. Anything flagged as incentivized is docked 0.38 of a star before it enters the pool — a figure chosen to sit just under the measured inflation rather than to punish. The distribution's shape is checked, so a suspicious J-curve or bimodal split is caught rather than averaged into a smooth-looking mean. A sudden spike in review velocity lowers our confidence in the product's data; it never moves the score by itself, because the alternative is a system that can be gamed by buying attention.

The pooled result is then shrunk toward the category: a product's rating is blended with its category's typical rating in proportion to how many reviews it actually has. This is the step doing quiet work in the cleanser example above. The INKEY List cleanser's handful of hundreds gets pulled meaningfully toward the category middle, because a few hundred people is a modest sample; the products with tens of thousands barely move, because that many people is a real measurement. Neither product is rewarded or punished for popularity — both are treated as samples carrying different confidence.

Finally, reviews age. Weight halves roughly every two years, and if a formula's ingredient fingerprint changes, the review pool resets entirely. A reformulated product does not get to keep a decade of praise earned by a formula it no longer contains.

The other 65%: what reviews say, not what they scored

The more interesting share of Factor 3 has nothing to do with stars.

A star is one number standing in for an entire experience, and it compresses away the thing that actually matters — whether the specific problem someone bought the product for got better, stayed the same, or got worse. So the majority of the results factor comes from the review text: sampled records classified into improved, no effect, and worsened, then scored on the balance between them, with no-effect counted as a partial negative rather than a neutral.

This needs a real sample to mean anything, so the text signal only engages once enough records exist for a product; below that threshold the factor falls back to stars alone and openly says so through a lower confidence band. Confidence here is a band, never a weight — thin data lowers what we claim, not what a product scores.

Two modifiers sit alongside it. Repurchase language — people saying they bought it again, that it is a permanent fixture — moves the factor within a bounded range, because repurchase is a behaviour rather than an opinion and behaviours are harder to fake. And a tolerability check counts how often irritation and breakout complaints appear, which functions as the field test on the chemistry: Factor 2 predicts from the ingredient list how likely a formula is to irritate, and this is where reality gets to disagree with the prediction. When a formula looks gentle on paper and the complaint pattern says otherwise, the complaint pattern is the more useful document.

That is why we treat a product's complaint clusters as more informative than its rating. Fifty people independently reporting pilling under sunscreen is a formulation fact. Fifty five-star reviews are a mood.

The ceiling, stated as arithmetic

Here is the part worth being concrete about, because "we weight it appropriately" is exactly the sort of claim that should be checkable.

The star signal is a third of a factor that is a fifth of the overall score — under a tenth of the final number, at absolute maximum, if a product went from the worst possible rating to the best. In the cleanser example, the entire star spread between those four products, once shrunk and put through those two multiplications, moves an overall score by under a single point. The thirty points actually separating them came from somewhere else entirely.

That is the whole answer to "why doesn't the most-loved product win." It cannot. Not because we overrode the crowd, but because the crowd's contribution is bounded by design and the other 80% of the score — formulation quality, safety and tolerability, usability, value, brand conduct — has considerably more room to move.

Two further limits matter. General sentiment is never treated as condition-specific evidence: a product being widely adored does not become proof that it works for rosacea, because the people writing about it were not asked about rosacea and were not randomised to anything. And brand fame is not scored at all — only countable events and verifiable credentials are. A product cannot earn a point for being famous.

The honest limitation of all this is that a de-biased average is still an average of a self-selected group. We can shrink it, age it, discount the paid reviews and cross-check it against the text. We cannot turn it into a trial. What we can do is stop it pretending to be one.

Reading the reviews yourself, better

If you take one habit from this, make it reading the shape of a rating rather than its headline.

Check how many reviews there are before you believe a decimal place. A two-tenths gap is meaningful across tens of thousands of ratings and close to noise across a couple of hundred — and the sites reporting those averages rarely distinguish. Look at the one- and two-star reviews specifically, not for the anger but for the pattern: repeated, specific complaints about pilling, stinging, scent or pump failure are the most reliable information in the whole pile, because those are the things a review can actually observe. Vague dissatisfaction tells you nothing; a dozen people describing the same texture failure tells you a great deal.

Check the date, too. A product reformulated eighteen months ago still carries every review of the version that no longer exists, and no retailer separates them for you.

When you want to see how two products differ across all six factors rather than on their star line, the compare tool lays the breakdown out side by side. And if the question is not "is this good" but "is this good for me" — which is usually the real question — a saved skin profile changes which grade leads on every page you visit, because a product's fit for your skin and its aggregate popularity are answers to entirely different questions.

The ratings deserve a seat at the table. They just don't get to run the meeting.