Critics’ scores: tipping the balance

With the recent addition of Jeannie Cho Lee MW as Wine Lister’s fourth partner critic, now seems a good opportunity to elucidate our approach to calculating Quality scores.

Wine Lister’s Quality scoring algorithm takes into account each critic’s rating scale and the relative generosity or severity of their scoring.

Vinous and Jeannie Cho Lee rate on a 100-point scale, whereas Jancis Robinson and Bettane+Desseauve rate on a 20-point scale.  But we also know that the former pair never (or very rarely) give a rating less than 70, while the latter never (or very rarely) give a rating less than 10.

The first step in our scoring algorithm is to put all ratings on a level playing field by rebasing them. We take each critic’s minimum and maximum wine ratings and spread these back out over our entire 100-point scale. The result is that, for example, a Bettane+Desseauve rating of 15 and a Jeannie Cho Lee rating of 85 both score 50 out of 100.

But it’s not quite that simple! We also know that the way the critics score is different. For example, we find that Jeannie Cho Lee tends to give fewer high scores than Vinous. Our algorithm takes into consideration these tendencies towards more or less generous ratings, and adjusts scores as necessary for a fair outcome. This is our score normalisation process.

Using nonparametric statistical techniques, for the same wine-vintages that the critics have rated, we look at each critic’s probability of awarding each score, and use the difference in probabilities to make necessary adjustments.

Beauregard 2009 Quality score blog

For example, we looked at the set of wine-vintages rated by both Jancis Robinson and Bettane+Desseauve, and found that wines are around 5% more likely to achieve a rating of 17 or higher from Bettane+Desseauve. Once normalised, this means that a wine such as Château Beauregard 2009, rated 17 by both Bettane+Desseauve and Jancis Robinson, achieves Wine Lister critic scores of 73 and 76 respectively. In other words, 17 is harder to get from Jancis Robinson than from Bettane+Desseauve. This is illustrated in the image above.

Once we’ve rebased and normalised the critics’ scores, we simply take an average of those scores, giving an equal rating to each of our four partner critics, each of whom represents one of the world’s key fine wine markets.

There’s still more… The Wine Lister Quality score also takes into consideration a wine’s ageing potential as defined by critics’ drinking windows – watch this space for a further explanation of how that works!


Jeannie Cho Lee MW becomes Wine Lister’s fourth partner critic

We are thrilled to announce the addition of Jeannie Cho Lee MW as Wine Lister’s fourth partner critic, representing the Asian market.

Wine Lister gathers data from multiple sources to assess thousands of wines on a truly holistic basis. Scores are made up of three main categories: Quality, Brand, and Economics. Critic ratings form a vital part of the Quality category, providing reliable scores and drinking window information.

Jeannie Cho Lee is an independent wine critic, a university professor, and an award-winning author who in 2008 was the first Asian to become a Master of Wine. Based in Hong Kong, she writes about wine on her website, www.jeanniecholee.com, and has also published several books, including Asian Palate.

Adding to our unique collaborations with three of the most reputable wine critics in the world’s major fine wine markets – Jancis Robinson (UK), Antonio Galloni (Vinous) (USA), and Bettane+Desseauve (France) – we are delighted to welcome Jeannie Cho Lee to represent the key Asian fine wine market. This careful addition will help us in our goal of providing ever more reliable and accurate ratings and analysis, while crucially maintaining the high standards of this elite pool of critics. Jeannie Cho Lee’s scores have been added to those of our existing partner critics, and fed into Wine Lister’s bespoke algorithm to produce a Quality score out of 1,000 points. Wine Lister takes each critic’s minimum and maximum wine ratings and spreads these back out over the entire scale. As each wine critic scores differently (not just on a different scale), we also account for the frequency of ratings and normalise scores for fair comparison. Each critic is weighted equally.

Website wine page image

You may notice that some scores have changed for wines that have not been rated by Jeannie. In order to meet our goal of having the most comprehensive and up-to-date information, and the most rigorous rating algorithm, we have increased the sample set of scores upon which we carry out the normalisation between critics. This now includes the 18 months’ worth of new critic scores since our launch.