The core problem is simple: a plain average ignores how many votes went into it. A 4.8 rating from 300 reviews is not the same as a 4.8 from 6 reviews. The first one you can trust. The second one — who knows.<\/p>\n
The solution comes from Bayesian statistics<\/a> (whatever that might mean in this context exactly), et voilà:<\/p>\n Johnson — 2.72<\/b> Somehow, either the tighter spread of Miller’s scores (SD of 0.58 vs. 1.58), or the smaller number of ratings (4 vs. 10), or both together refined<\/b> the average, giving Miller the edge by a few hundredths. The tie is broken.<\/p>\n Brace yourself. It’s gonna sting a little.<\/p>\n We assume<\/a> there are K<\/b><\/i> possible ratings, indexed by k,<\/b><\/i> where each rating is worth sk<\/sub><\/b><\/i> points (“1” = 1 point, “2” = 2 points, and so on). Given N<\/b><\/i> total ratings for each item, with nk<\/sub><\/b><\/i> ratings for each k,<\/b><\/i> we can compute the item’s score using the following formula:<\/p>\n Where zα\/2<\/sub><\/b><\/i> is the 1−α\/2<\/b><\/i> quantile of the normal distribution. The resulting score is the lower bound of a normal approximation to the Bayesian confidence interval for the average rating. Setting, say, α=0.10 (z=1.65), the score S<\/b><\/i> means that in 95% of cases, the item’s true average rating will be above S<\/b><\/i>.<\/p>\n In plain English: the “advanced” calculation gives you a conservative estimate of what the average rating would converge to if you had more data. And — as we saw with Johnson and Miller — it lets you rank items even when their plain averages are identical.<\/p>\n I took 2,000 ratings on a 1-to-5 scale across a set of regional sales offices and computed both the plain average and the “advanced” score for each one.<\/p>\n Flushing dropped from 2nd to 4th place. The likely reason: its 66 ratings don’t inspire enough confidence that the average is truly that high. In 90% of scenarios, its predicted rating is no better than 4.55 — roughly 4th place territory.<\/p>\n Tribeca, on the other hand, was formally 13th. But thanks to a solid base of 249 ratings, the model predicts — with 90% confidence — an average of at least 4.4, which bumps it up to 10th.<\/p>\n My impression is that the formula works most convincingly for short rating scales, like the 1-to-5 used here.<\/p>\n In any case, here is a Google Sheets workbook<\/a> — it should handle rating scales up to 100, import up to 10,000 rows of individual ratings, and lets you adjust the confidence level (90% in our example).<\/p>\n https:\/\/www.evanmiller.org\/ranking-items-with-star-ratings.html<\/a><\/p>\n Advanced rating calculator<\/a> in Google Sheets<\/p>\n",
"summary": "Two employees, same average rating — but should we trust 10 scattered scores the same as 4 consistent ones? Inspired by Evan Miller's work on ranking items with star ratings, this post walks through a Bayesian approach that accounts for both spread and sample size.",
"date_published": "2019-09-21T15:59:00+03:00",
"date_modified": "2026-04-12T03:29:09+03:00",
"tags": [
"Bayes",
"equations",
"Google Sheets",
"ratings",
"statistics"
],
"image": "https:\/\/ivanbaldin.blog\/pictures\/ratings00.png",
"_date_published_rfc2822": "Sat, 21 Sep 2019 15:59:00 +0300",
"_rss_guid_is_permalink": "false",
"_rss_guid": "3",
"_e2_data": {
"is_favourite": false,
"links_required": [
"jquery\/jquery.js",
"fotorama\/fotorama.css",
"fotorama\/fotorama.js"
],
"og_images": [
"https:\/\/ivanbaldin.blog\/pictures\/ratings00.png",
"https:\/\/ivanbaldin.blog\/pictures\/ratings01.png",
"https:\/\/ivanbaldin.blog\/pictures\/ratings02.png"
]
}
},
{
"id": "7",
"url": "https:\/\/ivanbaldin.blog\/all\/call-quality-audits-determining-your-sample-size\/",
"title": "Call Quality Audits: Determining Your Sample Size",
"content_html": " A common tool for assessing the quality of sales managers’ work is quality auditing of phone calls<\/b>, or “call monitoring.”<\/p>\n Suppose your goal is not simply to measure<\/b> the quality of phone calls, but to document growth<\/i> in that quality.<\/b> For example, you conducted training for your managers, introduced new incentives for meeting quality standards, or implemented some other intervention.<\/p>\n It’s reasonable to assume that an improvement in quality in the first random call recorded after the training won’t definitively prove that quality has improved across all calls. Likely, a second successful call won’t conclusively confirm the hypothesis either.<\/p>\n Thus, you’ll need to listen to — if not all calls — at least a sufficient number<\/b> of calls after your changes, and the number of calls you’ll need to audit can, in fact, be calculated precisely.<\/b><\/p>\n On page 35 of the paper “Introduction to Statistical Inference and Experimental Design<\/a>” I found a suitable example 1.9 and a formula for calculating such samples:<\/p>\n In this example, an improvement in ACT<\/a> math test scores from 24.5 (variance 8.2) to 26.0 points is considered, with α = 0.05 and power = 0.90.<\/p>\n For convenience, I assembled the formula in a Google Sheet:<\/p>\n
\nMiller — 2.63<\/b><\/p>\nThe formula behind the advanced rating<\/h2>\n
![\"\"](\"https:\/\/ivanbaldin.blog\/pictures\/ratings00.png\")
\n<\/div>\nThe advanced rating in action<\/h2>\n
![\"\"](\"https:\/\/ivanbaldin.blog\/pictures\/ratings01.png\")
\n![\"\"](\"https:\/\/ivanbaldin.blog\/pictures\/ratings02.png\")
\n<\/div>\nSee also<\/h2>\n
Calculating Sample Size for Quality Call Audits<\/h2>\n
![\"\"](\"https:\/\/ivanbaldin.blog\/pictures\/create-a-1080-1080-pixels-flat-style-cartoon-showi-(1).png\"){kind=tracking}
\n<\/div>\nCalculating Sample Size<\/h2>\n
![\"\"](\"https:\/\/ivanbaldin.blog\/pictures\/samplesize00.png\")
\n<\/div>\n
![\"\"](\"https:\/\/ivanbaldin.blog\/pictures\/samplesize03.png\")
\n