Is a comparison between different national championships really possible? No, even if the rules should be the same for everyone in all competitions, when the judges’ panels change, the marks also change, and someone takes the opportunity to make comparisons that mean little.

In an article that was shown to me by a friend I found this sentence:

Hanyu recently won the Japanese nationals, but his short program score was more than four points lower than Chen’s on Saturday.

True, the official result says that Hanyu deserved 111.31 points, Chen 115.39. Howewer, I didn’t just look at the numbers, I wondered if the way the judges assign the marks in the two competitions is comparable. This is why I checked the Japanese national championship, the US one, and while I was there also the Russian one. I only watched the men’s competition, if I had the time it would be nice to do this type of control also for the other disciplines, and maybe even look at some other nation.

To see if certain behaviors are repeated over time, I checked the last two Olympic four years, so from the 2014-2015 season. I did not consider the 2020-2021 season because the Autumn 2020 competitions are in fact local competitions and have no statistical significance. For each season I checked the top five skaters in their respective national championship. The skaters are these:

For each of them I checked the marks they received in the GOE and PCS, based on the data provided by SkatingScores. These, for example, are Hanyu’s data from the last two seasons:

What interested me were the numbers in the “Mean GOE Mark” and in the “Mean PCS Mark” columns. I didn’t choose accidentally to show you Hanyu’s data. His marks are a bit peculiar, and need one more explanation, but I start with the general check first. I checked every single season, the one I am showing you for my explanation is the 2021-2022 season of the five Japanese skaters. For now you must ignore the red numbers and the lines from 52 to 56.

For each competition held in the autumn I have transcribed in a table the Mean GOE (column BP for the short program and BR for free skate) and the Mean Mark in PCS (column BQ for the short program and BS for free skate). Hanyu’s international line is gray because he hasn’t skated in international competitions this season. Then I have transcribed the averages of all the Japanese skaters in the boxes at the right:

BY42 = average of the GOE in SP of international competitions
BZ42 = average of the PCS in SP of international competitions
CA42 = average of the GOE in FS of international competitions
CB42 = average of the PCS in FS of international competitions

The BT-BW columns are dedicated to the national championship. In line 43 of the BY-CB columns I have transcribed the average of the marks of the five skaters.

In line 44 I calculated the difference. For example, 2.56 (average of five skaters at the National competition) -2.12 (average of five skaters in all international competitions) =0.44, and this is the number I’m interested in (like the next 0.29, 0.96 and 0.44).

Why am I interested in these numbers? Because they indicate how much more generous the national judges have been to their skaters than the international judges. Doing the math on several skaters instead of just one, who may have had a bad program one day that distorts the statistics, shows me the general trend.

In the short program of the national championship, Japanese judges tended to give each element presented by the skaters a higher GOE of 0.44 than the marks that those skaters received in international competitions. These are numbers I can compare, and that’s exactly what I did.

I calculated the variation between the marks received by skaters in all international competitions (excluding the Japan Open) and those they received in the national championship for seven of the last eight seasons and entered the data in two graphs, one dedicated to the GOE and the other to the PCS.

Since the ISU changed the value of the GOE between the 2017-2018 season and the 2018-2019 season, I drew a vertical line to easily distinguish the GOEs assigned in the +3/-3 period and those assigned in the +5/-5 period.

The red and orange lines are beautiful, they are almost always the highest and, especially this year, they reached incredible peaks, even if the light green line eventually managed to fit between the two. What mean? It means that the most absurd marks in the GOE are awarded in the short program of the American championship. This has been true in six seasons out of seven. This year, the short program of the Russian national championship follows. In the past the marks had been much lower, in a couple of seasons they had even been lower nationally than internationally. Could this change be due to the fact that we are in an Olympic season? In third place, with values ​​so high that they were only surpassed this year by the Russian short program, there is the free program of the American national championship. I would say that in the United States, inflating the marks of their skaters, even in an absurd way, is a habit. And Japan? Their values ​​are quite often intertwined with the Russian ones, in this year’s short program it was the nation that has inflated their marks the least, in the free skate it is practically in line with Russia.

The PCS:

Nice, we have the lines grouped in such a way that they almost look like they belong to two different charts. Three of the four lines of Russia and the United States follow very similar paths, only the Russian free skate intertwines in a slightly different way. At least in the Men’s competition Russia does not inflate the GOEs so much, or not so often, but they work a lot on the PCS, especially in the free skate. The United States is more consistent, makes no distinctions and inflates any scores. As for Japan, they does not even dream of doing so: their two lines are always below the lines of the other two nations.

If you have noticed, on the last column of each graph appear two stars and two hexagons, in blue and purple. Let’s go back to the table that I have already published above, and which, in order not to drive you crazy, I propose again here:

I have calculated the national statistics on five skaters, but only four of them have participated in at least one international competition. In the skaters I checked this situation occurred three times: with Andrei Lazukin in the 2016-2017 season, with Daisuke Takahashi in the 2018-2019 season, and with Yuzuru Hanyu in the 2021-2022 season. All Lazukin’s national data is low and contributes to lowering the national average for Russia. Three of Takahashi’s four figures (the exception are the free program GOEs) are high and help raise the national average. Hanyu’s data is so different from everyone else’s that it completely skews the stat. In the free program Hanyu got average marks in the GOE above +3, which none of the other skaters did in any free skate, international or national makes no difference. In the short program his average marks are higher than +4, no one has obtained marks higher than +4, only once did Kagiyama obtain marks higher than +3. With the PCS the situation is similar. How can I put in the national average marks so much higher than the others without having an absurd result?

In rows 46-50 of columns BX-CB I calculated the Japanese national average based only on the other four skaters. As you can see, the national championship average is lower (row 49), and consequently the difference (row 50) is also lower. The values ​​you find in line 50 (+0.04, +0.04, +0.68 and +0.27) in the two graphs are indicated by the stars.

But I didn’t like this solution. I’ve always used five skaters, and even to use only four for one graph was a way to distort it. So I looked at Hanyu’s latest international competitions, the World Championship and the World Team Trophy 2021. They are competitions from last season, but they are competitions with international judges, and they are not that distant in time, so they are the best thing with which I could work.

In row 54 of the BO-BS columns I entered the data of the World Championship, in the row below the data of the World Team Trophy, and in row 56 I calculated Hanyu’s average. This allowed me to see the numbers that I highlighted in red. The averages are all lower than the marks Hanyu received in the national championship, but they are also higher than almost all the marks received by other Japanese skaters in international competitions. Only Kagiyama in his free program on two occasions received marks higher than Hanyu’s average marks. So these marks allow me to put Hanyu in the calculations and have numbers reasonably correct.

In row 54 of the BY-CD columns I have entered the average of the data present in rows 43-50 and 54-55. With the highest international average, the difference (row 56) is smaller. In the graphs I have indicated these values ​​with hexagons.

In the end, what do all of these numbers mean for skaters’ scores? I only did the math for Hanyu and Chen’s short program of the last national championship, the ones in which, according to Barry Wilner, who wrote the article, Hanyu scored four points less than Chen.

Assuming the judges inflated both skaters’ scores, I lowered them. For Hanyu I used the values related to the revised national average, with his scores of Spring 2021, therefore a lower vote of 0.17 units in each technical element and of 0.07 in each item of the components. On the left I have indicated the real data, with the totals highlighted in the yellow boxes. In the center there are all the marks (the column with the +1 GOE is not in the protocols, but I entered it because I needed it to calculate the final score), on the right I have indicated my calculations, with the values of the changes written in red and the new totals inserted in the yellow boxes.

With or without the judges’ help, Hanyu’s score changes little.

If I had used the values of the difference between national and international competitions without considering the World Champion and the World Team Trophy 2021, Hanyu’s TES would have dropped to 60.28 points, the PCS to 47.58 points, the TSS to 107.86 points.

This is Chen’s protocol, with and without the help provided by the judges:

I know, it is not possible to say that these are the correct numbers, but these scores are much closer to reality than what was done on the competitions than the scores awarded by judges who clearly have a established habit of arbitrarily raising the marks of their compatriots. If the judgments had been objective in both competitions, it would have been Hanyu who received the highest scores, by well over four points.

Given that some of the national judges also judge international competitions, that these scores are used by the international press for propaganda, and that the other international judges can be influenced by propaganda, perhaps it is time for the ISU to put a stop to absurd marks that are awarded in many – not all – national competitions.