Game difficulty, skill levels and ELO

[Goodspeed]

I read some Reddit comment the other day in some thread about game theory where someone made the point that the difficulty of a game can be estimated by taking the amount of "skill levels" there are, where a skill level is a rating point where one wins about 2/3 of the games against the previous level, and loses 2/3 to the next level.
Intuitively, this logic seems sound. The more skill levels there are, the more there is to learn.

This is convenient for games that use ELO ratings, because with ELO there are ~100 points between the levels. That is to say, a 2000 rated player will win 2/3 of their games against a 1900 rated player.

In AoE3, there seem to be about 16 levels. On simple ELO, ratings range from ~1200 to ~2700. On EP ELO they range from 1400 to 1900, but the ladder isn't settled yet. It will be interesting to see if eventually, there will also be 16 levels on this ladder.
AoE2 has 1 or 2 more levels it looks like, with ELOs ranging from 1000 to 2700. Could this be due mostly to the larger player base? (https://www.voobly.com/ladder/ranking/8/0#pagebrowser1)
Sadly, SC2 uses a different rating system (TrueSkill), for which it isn't known (to me) how many rating points (MMR) 2 players would have to differ for one of them to win 2/3 of the games. Maybe one of you knows?

We can test the theory using 3 strategy games with player bases and histories that are large/long enough for the ladders to be settled, where we kind of know the order of difficulty: Checkers, Chess, and Go.
As expected, Checkers has the smallest range, from ~1100 to 2300. ~13 levels.
Chess has a much wider range, from ~900 to 2800. ~20 levels. Interestingly, RTS like AoE are apparently almost as hard as chess.
Go doesn't use ELO ratings, but its rating system can be converted to ELO (they did the math). It would range from 100 to 3600, which means there are ~36 levels.

The theory also holds up considering the ELO range back in AoE3's "more competitive" days was similar. On the old ELO website, the range was iirc ~900 to ~2400. This would mean that the range of ELOs actually doesn't relate to the size of the player base, but purely to the difficulty of the game. And it supports the idea that the current top players would have done about equally well in those days. If the competition was really that much fiercer at the top back then, one would expect at least one extra level. I can't say for sure what the lower end of the ELO ladder was back then though, I guess it may have been 800.

It's a shame SC2 uses a different rating system. I'm hoping one of you who has more experience with the game and its community can help me find some info on this. Surely someone did the math.

Discuss?

[Kaiserklein]

This is interesting, but totally relies on the aoe3 Elo rating being accurate, which it really isn't, for multiple reasons. Idk about other games

[Goodspeed]

Why isn't it accurate? The range is actually the same as it was in 2008, when we still had the old ELO website.

[lejend]

A lot of people don't know this, but Goodspeed always plays games on the highest difficulty setting, which is one thing I like about him.

[Horsemen]

A lot of people don't know this, but Goodspeed always plays games on the highest difficulty setting, which is one thing I like about him.

I guess you could call him a Lejend

[Djigit]

bollocks

[lemmings121]

Discuss?

I'm bit late since I dont usually lurk on offtopic, but you got my atention here.

I found a guy on reddit that did the math for sc2, Acording to him, a 300mmr gives you 68,6% chance of winning. a little bit above your 66% metric but we can go with it.

rankedFTW list the ladder players ranging from 1k to 7k. (a few players bellow that, but its hard to judge if they are actually playing or just quiting games, also Serral recently got to 7550, meaning we could call him almost two tiers above anyone else)

6k/300 = 20. thats roughly 20 "skill levels" (+2 if you count serral)

playing sc2, this feels right, 300mmr is a significant gap. but for aoe i'm not sure 100 elo points is right. the Elo math says it is, but I dont feel like it when looking into my QS history. But I assume thats because qs gives you so many random matchups with crazy balance that upsets can happen more easilly.

sources:
https://www.reddit.com/r/starcraft/comments/6dn6jf/does_anybody_have_more_detailed_stats_on_mmr/
https://www.rankedftw.com/ladder/lotv/1v1/mmr/

[Goodspeed]

Awesome. Thanks, I knew someone must have done the math already but couldn't find it

[chris1089]

This is interesting, but totally relies on the aoe3 Elo rating being accurate, which it really isn't, for multiple reasons. Idk about other games

What would you need for an elo rating system to be accurate. Is it just the small playerbase and number of games played, or is it something to do with the nature of aoe3 specifically?

[Goodspeed]

EP ELO now ranges from 1344 to 2180 (from 1400 to 1900 at time of OP). Interestingly, the upper limit moved much more than the lower one.

[chris1089]

I think this can be attributed to the distribution of skill in on Esoc patch compared to re patch.
Because the player base on Esoc is more highly skilled, people like me (who have less skill but play on Esoc patch) lose lots of games. Because there is the alternative of playing on re patch (whilst still playing the same game, but with the side effect of being on a different elo system) and playing people of the same skill/ not getting bashed/ other psychological reasons, people stop playing on Esoc patch, preventing their elo from dropping.

[chris1089]

Hope the logic is discernable

[Goodspeed]

Yeah that makes sense

[gibson]

I think it changing was at least partially due to the formula change

[Kaiserklein]

Yeah, the old formula was too punishing, staying on top was almost impossible

[chris1089]

Ah, what are the 2 formulae?

[chris1089]

Did they just change the constant by which the elo difference is multiplied?

[Kaiserklein]

Yes, it used to be 32 and now it's 16, or something like that

[chris1089]

Can it be seen that the change in formula starts a new trend where the max elo starts to increase, or did these two things begin to occur at different times?
If they began close together, we can probably attribute the increase in max elo to the new formula.

[deleted_user]

The aoe3/chess parallels are great in number.

[Goodspeed]

Can it be seen that the change in formula starts a new trend where the max elo starts to increase, or did these two things begin to occur at different times?
If they began close together, we can probably attribute the increase in max elo to the new formula.

The formula change just made it generally more volatile with big differences in rating iirc, which means both extremes should have moved equally.

viewtopic.php?f=19&t=16443&hilit=rating&start=25
Read from Kaiser's post

Edit: Actually, after re-reading that, I think the change can be attributed mostly to the formula change. Because the biggest effect of the change seems to be that the higher rated player now appropriately gains rating after winning, instead of gaining much too little.

[chris1089]

I was looking through the EP elo chart today and concluded that it was getting close to a fair representation of skill amongst those who had played 10s-100s of games. I remember that EP elo meant almost nothing at the lower end of players: the players with less elo had just played more games. Now I feel it's beginning to look more representative of skill.
At the top end there it seems a bit ridiculous that Mitoe can go to no. 1 without playing any of the top 10 players. Edit it's actually top 20 (at least now their elo has been farmed)

[RefluxSemantic]

One of the problems with EP is that EP essentially has some isolated groups of players, formed by the pre-existing rank. People find matches based on their ranking, so what's happening is that captains-majors form a sort of subgroup, while higher ranked players do the same. The mixing between these groups might not be enough, which will cause ELOs to get messed up the further you move away from the actual top playerrs.

[chris1089]

Would quick search being the sole means of ranked games, and pairing being based on elo fix this?

[Goodspeed]

Does the DE ladder use ELO?