I was already checking closely the National Championships for some time when I came across this tweet:
If you enter in the tweet, the graphs are really interesting. And, graphs aside, who wrote the tweet mentioned a Christine Brennan’s article. Obviously I searched for the article:
Judges don’t do Ashley Wagner any favors at U.S. championships
Brennan notes that the PCS marks awarded to Ashley Wagner at the National Championship 2018 were not inflated compared to those she received not long before at Skate America.
Cheating, you say? No, more like home cooking with the referees in a basketball game. In the days of the old 6.0 system, it wasn’t uncommon to see a great like Michelle Kwan be sent off to an Olympics with a bouquet of 6.0s, just because, whether she always deserved them or not.
Oooooh… So marks are awarded just because? I did not know, I probably did not read carefully enough the rules published by the ISU.
Are the scores at the National Championships inflated? Not for everyone, and not always. In the years immediately following the birth of the ISU Scoring System, the judges had to get used to the new system and evaluate its potential, therefore I noticed less peculiarities. Not that there aren’t any, but there are less. Surely none impressive as this:
I really love Virtue/Moir, but these marks are perhaps a little biased. Without going to extreme cases like this, how are the marks assigned to the national championships?
My initial idea was to check the GOE and PCS of all the skaters who have made it to the world podium, or the Grand Prix final podium, for as long as this scoring code exists. Only the Men’s competitions, I don’t have enough time to look also at Women, Pairs and Ice Dance. Why these competitions? Whoever has been on the Olympic podium, at least in the last 4 editions, has also been on the world podium, so it is already included in the definition. As for the Grand Prix Final, the strongest skaters participate, while the same cannot be said of the continental championship. How many European skaters have been able to get on the world podium in the last 10 editions of the Men’s competition? Three, once each the Russians Artur Gachinski and Mikhail Kolyada, and four times the Spaniard Javier Fernandez, against three North Americans (the Canadian Parick Chan and the Americans Nathan Chen and Vincent Zhou, for a total of seven podiums) and eight Asians (Japanese Takahiko Kozuka, Daisuke Takahashi, Yuzuru Hanyu, Tatsuki Machida, Shoma Uno and Yuma Kagiyama, Kazakh Denis Ten and Chinese Boyang Jin, for a total of 23 podiums). More strong skaters could participate in the Four Continents Championship than in the European Championship, and if everyone was there, winning was more difficult. On the other hand, this is a competition often deserted by the strongest in the Olympic seasons. No, the variables are too many. The best of the first part of the season go to the Grand Prix final, regardless of their nationality, sometimes even more than three skaters for a single season, so a victory there is more difficult, and more meaningful, than in the continental championship.
The problem is that in SkatingScores, from which I got the data, there are only the protocols of the Japanese, Russian and US championships. I know how to calculate that data from a single protocol, but doing it takes time, so I checked some skaters but ignored others. And I need the original protocols, which I haven’t always been able to find. Surprisingly, I had no problems with the Japanese federation site, even if I don’t know the language, and I easily recovered the protocol of the 2009-2010 season, which is not on SkatingScores, while I went crazy with the Canadian federation site. For this reason I ignored Jeffrey Buttle, but also skaters from other nations such as Stephane Lambiel, or even Fernandez and Ten, and instead I dealt with, among others, Sergei Voronov, of whom I already had most of the data without having to calculate them. alone. To explain what I did, I use Evan Lysacek’s graphs.
For each skater I checked all the competitions in which he participated. I only excluded (for some of the other skaters) the Japan Open and the Team Challenge Cup, I just can’t consider them serious competitions, and I added the missing national championship data every time I found the protocols. The first graph is on to the GOEs of the short program. I have not looked at the points, those vary not only according to the grade of execution but also according to the element performed, a +2 for a quadruple jump is very different from a +2 for a spin. I looked at the marks, that is, the number of +3, +2, +1… assigned by the judges to each element. SkatingScores indicates this under the heading Mean GOE Mark (red circle).
I didn’t keep the number as it is but I turned it into a percentage, which made it possible for me to compare the GOEs of the + 3/-3 system with those of the +5/5 system. In practice, to that -0.57 I applied the formula = (coordinates of the excel file box / 3) * 100. From the 2018/2019 season instead of the 3 there is a 5. In this case the result was -19.00 so Lysacek got 19,00% negative GOE compared to the maximum GOE he could have achieved with the elements he presented. For PCS these problems are not there, so I kept the data (green circle) as I found them. This number is the average of all component marks, not excluding the highest and lowest marks. Using this number allows us to easily compare the marks of the short program with those of the free skate. I haven’t done it for now, but I may do it in the future.
My file contains all the data, what I show you are the final graphs. I divided the seasons with a vertical black line. When the ISU judging system was adopted, Lysacek was already a senior skater, in the case of the junior seasons the background of the chart is not white but gray. I considered that a skater belonged to the junior category even if he has participated in some senior competitions, regardless of their importance.
I highlighted the national championship with a different blue, in this way it is easy to see if that competition was evaluated in a very different way than the others. On the GOEs there can be high variations from one competition to another, a couple of wrong jumping elements are enough to see the GOE fall, but it is curious how at the 2017 National Championship Lysacek obtained much higher marks not only compared to the previous competitions, but also compared to the following ones.
Strange as it is, this is the only competition that stands out, and it can happen that on a single occasion a skater performs much better than usual. These are the PCS.
In this case the variations are minor, but it is normal for the marks in the components to behave like this. In both 2006 and 2008 he set his personal best in the PCS, but the growth is not so great as to be striking. One note: I know National Championship scores don’t count as personal bests. In this text, I use the expression only because it is convenient for explanations. To be precise I would have to specify “unofficial personal best” every time, a definition too long for my taste. Once we understand why, only in this text, I express myself like this, we can move on. The GOEs in the FS:
The 2007 edition also stands out in the free program, so I went to look at the protocols. To make the difference between that competition and the others aren’t fall, but the fact that at the international level the marks of Lysacek oscillate mainly between 0 and +1, in that national championship they oscillate between +1 and +2. Or Evan was able to give quality to his elements of a sudden, and then he forget how to do it, or the judges pushed him a little bit. The PCS:
In this case the 2006 marks in components are really very high, and the suspicion that they are not really justifiable in relation to the quality of the program is inevitable. Has he really grown by over 0.80 points from his best performance? It took him over three years to get back almost to that level in the international competitions. The scores from 2007 also stand out, albeit a little less.
Why do judges give marks this way? I have already mentioned, both in Italian and in English (and again in Italian), Thinking, Fast and Slow, by Daniel Kahneman. The book was written in English, but I have the Italian edition so my quote is in Italian. You can understand it easily with an automatic translator. If you decide to buy the book, which I recommend you to do, this passage is in the second part, EURISTICHE E BIAS, Chapter XI: Ancore.
Ad alcuni visitatori dell’Exploratorium di San Francisco furono rivolti i seguenti quesiti:
La sequoia più alta del mondo è alta più o meno di 365 metri?
Quale ritenete sia l’altezza della sequoia più alta del mondo?
L’«ancora alta» in questo esperimento era 365 metri. Per altri volontari, la prima domanda faceva riferimento a un’«ancora bassa» di 54 metri. La differenza tra le due ancore era 311 metri.
Come previsto, i due gruppi produssero stime medie molto diverse: 257 e 86 metri, con una differenza tra l’una e l’altra di 171 metri
In short, those who had a high number in mind thought the sequoias were much higher than those who had a low number in mind. So if the judges know that a skater has received 9.00 in skating skills in a recent competition, they will hardly give him 7.50, even if this is the mark he deserves. They will give it a mark that is around 8.50, and they will also be convinced that they were strict.
In his book Kahneman never wrote about figure skating, but the way the human mind works is the same in all fields. I know I have read on at least two occasions texts on figure skating in which the author clearly wrote that a certain skater had been awarded too high marks in the national championship with the express purpose of influencing international judges, but at the moment I do not remember where I read these things. If I remember I will add the quotes.
The US judges has inflated Lysacek’s GOE’s once in the short program and two in the free skate, and his components at least once in the free skate. However the international judges do not seem to have been influenced by these marks. Evidently at that moment the result of the national championship was not being watched carefully abroad. For Johnny Weir, the variations from the best score ever received are smaller, but perhaps in 2006 they had some influence, at least in the short program.
As you can see there is a section that does not contain any data. I decided to make it visible when a skater, for whatever reason, has completely skipped one or more seasons.
Why this different treatment for Lysacek and Weir? The help to Lysacek is not as striking as that reserved for other skaters that I will write about later, but he has received a little help, Weir not. The judges of those years were less shameless than some current judges, but there may also be a question of internal politics. In an interesting article, which does not talk about marks, probably correct, but about political decisions that led to the formation of the American Ladies’ team for Sochi in 2014, M.G. Piety notes that sometimes there are things more important to the federation than what the athletes did when they skated. It applies to the choice of team members, but it can also be applied to marks. Let’s not forget that the judges are chosen by their federation. This means that judges are perfectly aware of having to assign marks that the federation likes, otherwise they will no longer judge any competition, or at least they will not judge the most important competitions.
Let’s see what some of the other federations did. For France I watched three skaters, Brian Joubert, Florent Amodio and Kevin Aymoz. Joubert:
The 2006-2007 and, a little less, 2007-2008 Nationals, stand out.
Compared to his usual, in December 2010 Joubert must have managed to perform his elements really well. However, this only happened in the GOEs, not in the PCS, and the international judges do not seem to have been influenced by these marks. Amodio:
In most cases it did not work, but with Amodio the French federation seems to have seriously committed to inflating both the technical and the artistic score. Aymoz:
I do not consider the national championship of the 2014-2015 season. Before, he made only one international competition, and a competition can go wrong. In my opinion, it makes no sense to make comparisons with previous marks and wonder about any anomalous growths if the skater has not competed at least in three international competitions. In the 2016-2017 season, however, the PCS of his short program grew a lot, in the 2018-2019 season they did it even more, what happened in the GOE is incredible, and this time it seems that the international judges have taken note and have adapted their marks.
When one of the skaters did not participate in a program – Aymoz did not qualify for the 2020 European Championship free skate – I left a gap. On the one hand this reminds us that he still participated in the competition, even if not in that program, on the other hand, if I had to decide to use these lists for something else, I could easily put side by side the scores of the two programs.
One of the skaters I have looked at is Michal Brezina. Because he? Because he has been active for many years, so it is possible to see the evolution of the score over a very long time, and because he skate for a federation not so influential.
For some national championships I have indicated the PCS but not the GOE. It happened with Brezina in 2007 and 2011, but also with someone else. Unfortunately in those cases I did not find the protocol but only the Detailed Result of the segment of the competition. The mark that I indicate is not the average of all the marks assigned by the judges, but the average of the final five marks. I have indicated these competitions with an asterisk.
One thing that stands out is that for some skaters the marks in the components do not change much, between the beginning and the end of the graph there is a point, a little more, a little less (Weir, Joubert, Brezina if we do not consider the junior period ), others have much more pronounced growths, with Aymoz’s being the most notable so far.
For Canada I watched two skaters and half: Patrick Chan, Nam Nguyen and Keegan Messing. Messing started to compete for US, then skipped to Canada, so he partecipate in two different national championship
Oh, this graph is really beautiful. Sudden growth of marks in 2009 and 2011, with subsequent international marks that seems to have been influenced by the national ones, very high peaks, but with no significant international effects, in 2012 and 2016. I suppose that the Canadian judges have perfectly understood how the system works. It is not for nothing that they were the ones who have assigned the perfect score to Virtue/Moir.
Impressive growth also in the PCS.
I only notice one detail: the most impressive peak, the first time that Chan had average ratings above +2 (it would be 66,66%, of the maximum score obtainable with his layout, instead he got 73,67%) it occurred only two months before his first success at the World Championship. When he won the world title, his percentage was 51,57%, much lower than the national championship but much better than his best international competition, the Four Continents Championship 2009, where he had taken 42,33% of the GOE at his disposal.
The following year the Canadian judges were equally generous, and again the competitions after the national championship seem to have been affected by those scores. It could be a case or not, certainly Chan has not arrived often at that level in international competitions.
I avoid further comments, so much we understand the way the Canadian judges assigned the marks to Chan. To Chan, not to all. I wanted to watch Kevin Reynolds too, I gave up because every skater takes time, especially if the results of his national championships don’t appear on SkatingScores. This is Nam Nguyen:
Nothing relevant in the GOEs, two PCS interesting, but not so interesting as Chan’s.
In the FS Nguyen was aided more than the in the SP. For Messing we need to remember that for the first seasons he was American.
The scores for Messing don’t seems strange, little variations compatible with a good competition.
For China I watched only Boyang Jin.
Two clarifications. The first concerns the column in a different blue, which indicates last season’s Grand Prix competition. In theory it was an international competition, but if you skate in China, you are Chinese, all your opponents are Chinese and all the judges as well, I can hardly think of it as an international competition. Nothing against China, I think the same for the Russian skaters and the Rostelecom Cup, the Japanese and the NHK Trophy and the Americans and Skate America. It would be much more dignified for the ISU to define those competitions as simple international competitions, but I don’t think the ISU has very clear ideas about dignity. So I recorded these scores, but I don’t take them very seriously.
Secondly, in my charts there is not this season. I started working on this graphs well before the start of the season, if I had to update them I would need time, in the meantime there would be other competitions, I would have to update again… and I would only be able to keep up with the competitions next summer. No, it’s easier if I ignore this season.
Only one attempt to help Jin in the components, at least if we exclude the last Cup of China, which in any case had no international effects.
I’m sorry I wasn’t able to find the protocol of the 2018-2019 season, with Jin returning from a bad World Championship but also from a fourth Olympic place. The marks stands out in only one competition, in his last year as a junior, but it does not seem that those marks have influenced the international judges. Could it be because before Jin the only Chinese skaters able to get on the world podium were Lu Chen among Ladies in the 90s and in more recent years some Pairs?
For Italy I watched only Mattero Rizzo and Daniel Grassl. Rizzo won a bronze at the European Championship, Grassl was only fourth, even if he has the potential to win some medal in the coming years. In theory I should have ignored them, but to me it did seem right to give Italy some space.
I considered Rizzo a junior skater in the 2017-2018 season because he competed in two junior Grand Prix competitions, winning one, and in the Junior World Championship, winning bronze. That year the nineteen year old Matteo participated in many competitions such as the Olympic Games (and before going to PyeongChang he participated in the Nebelhorn Trophy to get the qualification) and in the European and World Championships. He is the most striking case of a skater who competed in both categories at the same time. The National Championship of December 2017 stands out, and it is an important championship. The defending champion was not Rizzo, but Ivan Righini, and it was necessary to determine who would leave for Korea. Too bad I don’t have the time to look at Righini’s scores too.
Nothing strange in the free skate GOE, only one edition, that of 2016-2017, which stands out in the PCS, overall nothing really striking. Daniel Grassl:
To make these graphs, I had to compile long lists of data. I don’t publish them just because it would be a long job. However, I found it useful to gather the most sensational data in a table. I have marked all the occasions in which in a national championship the GOEs have grown by 10% or more compared to the best international competition already disputed, as long as the skater in question has already participated at least at three international competitions. Likewise, I have marked all the occasions in which the marks in the components have grown by 0.50 points or more compared to the best international competition.
In this table there are no American skaters, to which I have dedicated a special table, just as I have dedicated one to Japanese skaters and one to Russian skaters. For Messing the most striking data is linked to the period in which he competed for the United States, but since I have already published his graphs I have entered him in this table.
For each of the competitions that should be watched carefully I have indicated which season the national championship refers to, the percentage of GOE, or the highest scores in the PCS, and the best previous international competition with the relative percentage, or the relative PCS. The last box, colored, contains the difference between the two data.
The colors help us understand how much skaters have improved in the single national championship. Considering that for the GOE the large variations are frequent because they are linked to the eventual success of the jumps, I considered as worthy of attention only the increases above 10,00%. This does not automatically mean that those marks are inflated, but it is possible that they are, and if this type of growth is repeated on multiple occasions it is even more likely that the marks are inflated. Yellow is for GOEs that grow between 10,00 and 14,99%, orange for those who grow between 15,00 and 19,99%, red for those who grow between 20,00 and 24,99%, purple for those that grow by over 25,00%, a sign that the judges were really shameless.
For the PCS I have indicated in yellow the variations between 0.50 and 0.74 points, in orange those between 0.75 and 0.99 points, in red those between 1.00 and 1.24 points and in purple those above 1.25 points. I acknowledge that I have set the numbers arbitrarily, but if in most of the cases the judges of the national championships have remained below these values and therefore do not appear in my table, it means that these numbers can be taken seriously. When the judges exceed these limits, either the skater has proposed an extraordinary performance or the judges have inflated his score. I can believe in an extraordinary performance (even if, as I have written many times, every competition should be watched carefully). With two extraordinary performances I’m a bit wary. If the extraordinary performances are three, my confidence in those judges drops dramatically.
Which judges have repeatedly found their skaters extraordinary? The French ones with Florent Amodio, three times in the short program PCS, four in the free skate, but also with Kevin Aymoz, four times in the free skate, only once, but with an incredible growth, in the short program GOE. Worse did the Canadians with Patrick Chan, whose scores were often inflated not only in the PCS but also in the GOE, which is much rarer, and two times the inflation is really high. Messing and Nguyen did not receive the same courtesy, perhaps because world medals were not expected from them. Little to note for smallest or less influential federations: the Chinese judges with Jin, Czech judges with Brezina, French judges with Joubert and Italian judges with Grassl and Rizzo (perhaps for Italian judges I should watch Ice Dance) seems mostly correct.
Do you know a thing? I wrote too much. So to the Russian, Japanese and American skaters I’ll dedicate another post.
For Brezina some of the Nationals were the ones held with Slovakia, Poland and Hungary and therefore having more mixed panels? I can’t remember when they started to hold joint Nationals.
“The first concerns the column in a different blue, which indicates last season’s Grand Prix competition. In theory it was an international competition, but if you skate in China, you are Chinese, all your opponents are Chinese and all the judges as well, I can hardly think of it as an international competition. Nothing against China, I think the same for the Russian skaters and the Rostelecom Cup, the Japanese and the NHK Trophy and the Americans and Skate America. It would be much more dignified for the ISU to define those competitions as simple international competitions, but I don’t think the ISU has very clear ideas about dignity. So I recorded these scores, but I don’t take them very seriously.”
I wouldn’t consider them either.
The mimimum (for lower tier of international competitions, those below CHallenger Series status) is:
Technical panel from 3 different ISU members. International qualification needed (in exceptional cirmustances a technical specialist without international qualification can be used);
panel of judges of minimum 5 people with no ISU members having a majority of the panel. So 2 judges from organizing country out of 5 is fine but not 3 out of 5.
Even though I had to look up all the protocols for the Czech championship, I didn’t check where the competitions took place or how the judges’ panels were composed. Brezina’s marks interested me because he has really been competing for many years, but yes, his national championship is a bit strange. I might decide to check the trend of the seasonal average score year after year, to see if anyone has abnormal growths, but I haven’t decided yet.
If SkatingScores had had the data of some other national championship I would have done a few more charts, but looking for the protocols and doing the calculations is a long and tedious job.
I just looked: in China all were Chinese except one from Hong Kong, Technical Controller in the Men’s competition, judge in the Ladies and Pairs. In the United States, all Americans. In Russia, all Russians except Jeroen Prins, Technical Controller in Pairs (what was Prins doing in Russia? I think I will look at him better, also because he will be in Beijing) and the Ukrainian Sergei Baranov, Technical Specialist in Ice Dance. In Japan all Japanese. I would say that these competitions lack the basis to qualify as international competitions.
As for the participants, in China in four disciplines there are 19 Chinese skaters in all, in the United States 34 from the United States and 6 from other nations, in Russia 27 Russians and 13 non-Russians, this is the most international competition of all. In Japan, with only three disciplines, 24 Japanese and one Korean participants. These competitions were little more than anticipations of their respective national championships.