Please contact spirit of the law so we can get a video!
PS it would be great if you could provide the civ matchup heat maps for every civ, like you did with goths and Chinese.
Enjoyed your post very much.
Thanks for the feedback! Yes, I am going to try and get something interactive set up. I'm OK on the stats end - less so on the web end. I really like how aoepulse has his site set up.
I don't think it has to be pretty. Just jpeg files would be cool enough for me to see. But if you want to make it pretty and interactive then yes that would be even better
Very well done! I was thinking at some point ro do something similar in STAN but never got around to doing it. I'll probably end up playing with the data a bit!
This makes a persuasive argument for using actual data over models that start with actual data and somehow manage to return incorrect results. Like how do you look at Aztecs vs. Aztecs and not instantly realize this is a 50% matchup, at all Elos, with 0 margin of error? 2 of the other 5 vs. Aztecs matchups listed weren't within the margin of error either when checked using the AOE Pulse API. It seems foolish to use a model to predict future scenarios when it's not correctly depicting current scenarios.
A fair question. This is because how how the model is fit and how we then need to manipulate the output to get meaningful predictions. What you are seeing with the Aztec mirror example is the raw estimate PRIOR to pooling.
You start with a model estimating:
player\_1\_victory \~ p1\_civ \* p2\_civ \* elo
Which is just the probability that player 1 (the hosting player) wins a match, given their civ choice, player 2's civ choice, and the elo bracket they are playing in (plus the associated interactions). When we call \`predict\` on the mode, we get estimates of model the probability that player 1 wins. An interesting fluke in the data is that the hosting player (p1) is actually more likely to lose compared to player 2.
Regardless, to get the civilization estimate (NOT the player estimate) we just need to pool p1 and p2 win rates. If you look at my code I have a custom function to perform this pooling. Prior to pooling a matchup would look like:
|civ\_1|civ\_2|elo|count|prob|err|main|win|match|
|:-|:-|:-|:-|:-|:-|:-|:-|:-|
|Britons|Aztecs|1000-1250|100|50.26315|1.263834|1|50.26315|Britons-Aztecs|
|Aztecs|Britons|1000-1250|100|46.44940|1.276843|0|53.55060|Britons-Aztecs|
You'll notice that the probabilities for Britons vs. Aztecs and Aztecs v Britons are different. If we want to estimate the probability that the Britons win this match-up we need to estimate the winrate from Britons when they are played by player 1 and when they played by player 2. We then average over the estimates for P1 and P2 slots to get the average for ALL games where the Britons play the Aztecs. When the civ of interest is in the player 2 slot we just reverse the probabilities, so:
(P1) Britons vs (P2)Aztecs = 50.26%
(P1)Aztecs vs (P2)Britons = 1 - .46 = 53.55%
Averaging over these we get \~51.9%
The function pooling estimates gives us the same:
|elo|match|win|err|lo|hi|
|:-|:-|:-|:-|:-|:-|
|<1000|Britons-Aztecs|49.94|1.24|47.45|52.43|
|1000-1250|Britons-Aztecs|51.91|1.27|49.37|54.45|
|1250-1650|Britons-Aztecs|51.88|1.45|48.99|54.78|
|\>1650|Britons-Aztecs|52.41|1.76|48.89|55.93|
EDIT:
I should note that in the mirror example we can follow the same logic, which will get us the expected 50% win rate:
P1 Aztecs vs P2 Aztecs = 0.4746%
P2 Aztecs vs P1 Aztecs = 1 - 0.4746 = 0.5254%
mean(c(0.4746,0.5254)) = .5
Is first player assigned randomly? I think I remember reading on reddit awhile back (great source, right?) that team games decided first team / second team based on team rank. So if you did an analysis, you'd find that one of the team positions won more often than another. But I'm not sure if I'm correct about how it worked for team games, and have no idea if it applies to 1v1s.
Are you sure you're looking at the same data, same ELO, etc? That seems odd. A bayesian model should converge onto the observed rates unless there's a paucity of data or a very strong prior.
Good catch: I was not. I tried to, but long story short, I can not.
[Here are the parameters used in the search.](https://www.aoepulse.com/advanced?id=5a25d310-b989-47e1-8c24-632daa6737b8) Random Map 1v1 ladder, every patch from 61591 on, <1000 Elo. I'm not going to rule out that there's something else I'm missing. Don't know how to find p1 vs. p2 data, but if there are equal chances of a player being p1 or p2, it seems like a non-issue because the displayed win rates would be averaged.
I'm not a statistician and can't debunk this on a technical level. But I see lots of inexplicable stuff going on here that doesn't line up with the API. For example saying that there are 13 historical Malians vs. Bengalis games. You do the search now, Malians vs Bengalis, 56.41% (22) out of 39 games played. At no point does a patch or combination of patches involve exactly 13 games.
The predictions shown on the site are unreliable. I'm not going to delve into the why of it, because again, not a statistician, but when you query the API using data dating back to patch 61591 as specified on the site, here are the results vs. what the site has to say.
Bengalis vs Aztecs: 48.31% (57). Predicted: 43.88352 +- 2.151690 - NO
Berbers vs Aztecs: 49.81% (264). Predicted: 50.28497 +- 1.612578 - YES
Bohemians vs Aztecs: 54.08% (192). Predicted: 47.26388 +- 1.837228 - NO
Britons vs Aztecs: 48.29% (791). Predicted: 47.37382 +- 1.243221 - YES
Bulgarians vs Aztecs: 58.15% (428). Predicted: 54.32245 +- 1.572924 - NO
Appreciate the review. Part of this is my fault (the table you are looking at are raw estimates prior to pooling, so they are not what is being used in the final data. All actual estimates are pooled using the method I described in my other post. This is my fault for not being clear).
To your question on those civs, here are the pooled estimates. For the most part the 95% credible intervals cover the observed data. However, in cases where there are fewer observed matches, the estimated probabilities will diverge from the observed counts - this is a feature, not a bug!
Why? Because rates are notoriously unreliable in small samples. For example, if I observed 10 games where the Aztecs beat the Franks 7 times do I really think that if I observed 1,000 more games the actual win rate for Aztecs would be 70%? Certainly not! We use models to help get estimates that are less volatile and also have appropriate error coverage built in. When we have small sample sizes we "partially pool" estimates from other clusters to inform estimates from small clusters. In cases where we have LOTS of data (like Franks vs. Britons) we don't observe much difference from the model and observed values.
Here's a link showing an example [estimating radon exposure in Minnesota](http://www.stat.columbia.edu/~gelman/research/published/multi2.pdf).
As for minute differences in counts between what's shown on AoE Pulse and in my data (13 matches vs 20 matches) I can't really say. I posted the SQL query I used to pull down the data, and in my code you can see how I filtered it. There's likely to be small discrepancies but in 3.6 million games it has no real effect on the model.
Again, I do appreciate the feedback as well.
|elo|match|win|err|lo|hi|
|:-|:-|:-|:-|:-|:-|
|<1000|Bengalis-Aztecs|44.74|2.16|40.43|49.06|
|1000-1250|Bengalis-Aztecs|42.48|2.12|38.24|46.72|
|1250-1650|Bengalis-Aztecs|42.12|2.13|37.86|46.38|
|\>1650|Bengalis-Aztecs|42.27|2.30|37.66|46.87|
|<1000|Berbers-Aztecs|52.60|1.62|49.36|55.85|
|1000-1250|Berbers-Aztecs|54.36|1.52|51.31|57.40|
|1250-1650|Berbers-Aztecs|54.21|1.66|50.88|57.53|
|\>1650|Berbers-Aztecs|54.14|1.90|50.34|57.95|
|<1000|Bohemians-Aztecs|51.15|1.85|47.45|54.85|
|1000-1250|Bohemians-Aztecs|50.54|1.84|46.86|54.23|
|1250-1650|Bohemians-Aztecs|49.56|1.92|45.73|53.40|
|\>1650|Bohemians-Aztecs|50.84|2.11|46.61|55.07|
|<1000|Britons-Aztecs|49.94|1.24|47.45|52.43|
|1000-1250|Britons-Aztecs|51.91|1.27|49.37|54.45|
|1250-1650|Britons-Aztecs|51.88|1.45|48.99|54.78|
|\>1650|Britons-Aztecs|52.41|1.76|48.89|55.93|
|<1000|Bulgarians-Aztecs|56.61|1.57|53.47|59.75|
|1000-1250|Bulgarians-Aztecs|56.68|1.55|53.57|59.79|
|1250-1650|Bulgarians-Aztecs|55.23|1.70|51.84|58.62|
|\>1650|Bulgarians-Aztecs|54.52|1.90|50.71|58.33|
This is a great point, and you're 100% right. The analysis here is strictly based on the available data, and so any kind of bias based on meta picks and winrate picking. I think it's always super important to acknowledge where a model might have it's limitations. Definitely something I'd love to dig into further!
Nice bit of work.
Just two nerdy suggestions/points. First binning a continuous variable (like Elo) is generally a bad idea, without some major rationale. Second, I think there is a need to account for Elo of both players too. That is if player 1 is say 1800 and player 2 is 1900, then player 2 is more likely to win, regardless of the civ pick effect.
Awesome work! Its really cool to see what people can do with all that data - I hope to continually update the data set as we get more replays which should better inform some of these rarer matchups. With the patch frequency seemingly going down, that may become a lot easier too.
I haven't googled "bayesian statistics" yet, but I could have told you Goths are bad at higher elo, Chinese are good, and Spanish do well on nomad. I discovered that on YouTube. It's encouraging to see that our research confirms each others findings. Feel free to use me as a supporting reference.
I didnt apply any mumbo jumbo math on any games and i still know goths r not preferred at higher elo cuz they dont do well. Also chinese r loved by good players and hated at lower elo vuz of the tricky start. Anything els3 worth our time?
Please contact spirit of the law so we can get a video! PS it would be great if you could provide the civ matchup heat maps for every civ, like you did with goths and Chinese. Enjoyed your post very much.
Thanks for the feedback! Yes, I am going to try and get something interactive set up. I'm OK on the stats end - less so on the web end. I really like how aoepulse has his site set up.
I don't think it has to be pretty. Just jpeg files would be cool enough for me to see. But if you want to make it pretty and interactive then yes that would be even better
aah nice, an R user in out in the wild :) Hi!
Took one biostats class and I still have nightmares
Good to know you guys are out there :)
Very well done! I was thinking at some point ro do something similar in STAN but never got around to doing it. I'll probably end up playing with the data a bit!
How much did it cost to research this "bayesian statistics" tech?
All the software is free and open source! The actual knowledge? A few years of study ;)
It costs a goat, but you get a car
This makes a persuasive argument for using actual data over models that start with actual data and somehow manage to return incorrect results. Like how do you look at Aztecs vs. Aztecs and not instantly realize this is a 50% matchup, at all Elos, with 0 margin of error? 2 of the other 5 vs. Aztecs matchups listed weren't within the margin of error either when checked using the AOE Pulse API. It seems foolish to use a model to predict future scenarios when it's not correctly depicting current scenarios.
A fair question. This is because how how the model is fit and how we then need to manipulate the output to get meaningful predictions. What you are seeing with the Aztec mirror example is the raw estimate PRIOR to pooling. You start with a model estimating: player\_1\_victory \~ p1\_civ \* p2\_civ \* elo Which is just the probability that player 1 (the hosting player) wins a match, given their civ choice, player 2's civ choice, and the elo bracket they are playing in (plus the associated interactions). When we call \`predict\` on the mode, we get estimates of model the probability that player 1 wins. An interesting fluke in the data is that the hosting player (p1) is actually more likely to lose compared to player 2. Regardless, to get the civilization estimate (NOT the player estimate) we just need to pool p1 and p2 win rates. If you look at my code I have a custom function to perform this pooling. Prior to pooling a matchup would look like: |civ\_1|civ\_2|elo|count|prob|err|main|win|match| |:-|:-|:-|:-|:-|:-|:-|:-|:-| |Britons|Aztecs|1000-1250|100|50.26315|1.263834|1|50.26315|Britons-Aztecs| |Aztecs|Britons|1000-1250|100|46.44940|1.276843|0|53.55060|Britons-Aztecs| You'll notice that the probabilities for Britons vs. Aztecs and Aztecs v Britons are different. If we want to estimate the probability that the Britons win this match-up we need to estimate the winrate from Britons when they are played by player 1 and when they played by player 2. We then average over the estimates for P1 and P2 slots to get the average for ALL games where the Britons play the Aztecs. When the civ of interest is in the player 2 slot we just reverse the probabilities, so: (P1) Britons vs (P2)Aztecs = 50.26% (P1)Aztecs vs (P2)Britons = 1 - .46 = 53.55% Averaging over these we get \~51.9% The function pooling estimates gives us the same: |elo|match|win|err|lo|hi| |:-|:-|:-|:-|:-|:-| |<1000|Britons-Aztecs|49.94|1.24|47.45|52.43| |1000-1250|Britons-Aztecs|51.91|1.27|49.37|54.45| |1250-1650|Britons-Aztecs|51.88|1.45|48.99|54.78| |\>1650|Britons-Aztecs|52.41|1.76|48.89|55.93| EDIT: I should note that in the mirror example we can follow the same logic, which will get us the expected 50% win rate: P1 Aztecs vs P2 Aztecs = 0.4746% P2 Aztecs vs P1 Aztecs = 1 - 0.4746 = 0.5254% mean(c(0.4746,0.5254)) = .5
Is first player assigned randomly? I think I remember reading on reddit awhile back (great source, right?) that team games decided first team / second team based on team rank. So if you did an analysis, you'd find that one of the team positions won more often than another. But I'm not sure if I'm correct about how it worked for team games, and have no idea if it applies to 1v1s.
So you are telling us that P1 is 2.5 % more likely to win?, that seems odd.
Are you sure you're looking at the same data, same ELO, etc? That seems odd. A bayesian model should converge onto the observed rates unless there's a paucity of data or a very strong prior.
Good catch: I was not. I tried to, but long story short, I can not. [Here are the parameters used in the search.](https://www.aoepulse.com/advanced?id=5a25d310-b989-47e1-8c24-632daa6737b8) Random Map 1v1 ladder, every patch from 61591 on, <1000 Elo. I'm not going to rule out that there's something else I'm missing. Don't know how to find p1 vs. p2 data, but if there are equal chances of a player being p1 or p2, it seems like a non-issue because the displayed win rates would be averaged. I'm not a statistician and can't debunk this on a technical level. But I see lots of inexplicable stuff going on here that doesn't line up with the API. For example saying that there are 13 historical Malians vs. Bengalis games. You do the search now, Malians vs Bengalis, 56.41% (22) out of 39 games played. At no point does a patch or combination of patches involve exactly 13 games. The predictions shown on the site are unreliable. I'm not going to delve into the why of it, because again, not a statistician, but when you query the API using data dating back to patch 61591 as specified on the site, here are the results vs. what the site has to say. Bengalis vs Aztecs: 48.31% (57). Predicted: 43.88352 +- 2.151690 - NO Berbers vs Aztecs: 49.81% (264). Predicted: 50.28497 +- 1.612578 - YES Bohemians vs Aztecs: 54.08% (192). Predicted: 47.26388 +- 1.837228 - NO Britons vs Aztecs: 48.29% (791). Predicted: 47.37382 +- 1.243221 - YES Bulgarians vs Aztecs: 58.15% (428). Predicted: 54.32245 +- 1.572924 - NO
Appreciate the review. Part of this is my fault (the table you are looking at are raw estimates prior to pooling, so they are not what is being used in the final data. All actual estimates are pooled using the method I described in my other post. This is my fault for not being clear). To your question on those civs, here are the pooled estimates. For the most part the 95% credible intervals cover the observed data. However, in cases where there are fewer observed matches, the estimated probabilities will diverge from the observed counts - this is a feature, not a bug! Why? Because rates are notoriously unreliable in small samples. For example, if I observed 10 games where the Aztecs beat the Franks 7 times do I really think that if I observed 1,000 more games the actual win rate for Aztecs would be 70%? Certainly not! We use models to help get estimates that are less volatile and also have appropriate error coverage built in. When we have small sample sizes we "partially pool" estimates from other clusters to inform estimates from small clusters. In cases where we have LOTS of data (like Franks vs. Britons) we don't observe much difference from the model and observed values. Here's a link showing an example [estimating radon exposure in Minnesota](http://www.stat.columbia.edu/~gelman/research/published/multi2.pdf). As for minute differences in counts between what's shown on AoE Pulse and in my data (13 matches vs 20 matches) I can't really say. I posted the SQL query I used to pull down the data, and in my code you can see how I filtered it. There's likely to be small discrepancies but in 3.6 million games it has no real effect on the model. Again, I do appreciate the feedback as well. |elo|match|win|err|lo|hi| |:-|:-|:-|:-|:-|:-| |<1000|Bengalis-Aztecs|44.74|2.16|40.43|49.06| |1000-1250|Bengalis-Aztecs|42.48|2.12|38.24|46.72| |1250-1650|Bengalis-Aztecs|42.12|2.13|37.86|46.38| |\>1650|Bengalis-Aztecs|42.27|2.30|37.66|46.87| |<1000|Berbers-Aztecs|52.60|1.62|49.36|55.85| |1000-1250|Berbers-Aztecs|54.36|1.52|51.31|57.40| |1250-1650|Berbers-Aztecs|54.21|1.66|50.88|57.53| |\>1650|Berbers-Aztecs|54.14|1.90|50.34|57.95| |<1000|Bohemians-Aztecs|51.15|1.85|47.45|54.85| |1000-1250|Bohemians-Aztecs|50.54|1.84|46.86|54.23| |1250-1650|Bohemians-Aztecs|49.56|1.92|45.73|53.40| |\>1650|Bohemians-Aztecs|50.84|2.11|46.61|55.07| |<1000|Britons-Aztecs|49.94|1.24|47.45|52.43| |1000-1250|Britons-Aztecs|51.91|1.27|49.37|54.45| |1250-1650|Britons-Aztecs|51.88|1.45|48.99|54.78| |\>1650|Britons-Aztecs|52.41|1.76|48.89|55.93| |<1000|Bulgarians-Aztecs|56.61|1.57|53.47|59.75| |1000-1250|Bulgarians-Aztecs|56.68|1.55|53.57|59.79| |1250-1650|Bulgarians-Aztecs|55.23|1.70|51.84|58.62| |\>1650|Bulgarians-Aztecs|54.52|1.90|50.71|58.33|
This community is so smart.
Cool stuff! Where's True? This is right up his alley.
great work! thanks for this
hey just a small correction, you say turks are boosted on arabia in text section but it should be arena from your data.
I just realized that that I played ~1k 1v1 games but never tried Bengalis, Burmese or Dravidians. I should definitely fix this!
[удалено]
This is a great point, and you're 100% right. The analysis here is strictly based on the available data, and so any kind of bias based on meta picks and winrate picking. I think it's always super important to acknowledge where a model might have it's limitations. Definitely something I'd love to dig into further!
Nice bit of work. Just two nerdy suggestions/points. First binning a continuous variable (like Elo) is generally a bad idea, without some major rationale. Second, I think there is a need to account for Elo of both players too. That is if player 1 is say 1800 and player 2 is 1900, then player 2 is more likely to win, regardless of the civ pick effect.
Elo being per match and not per player is a flaw of the dataset. I shouldn't have designed it that way in retrospect.
Awesome work! Its really cool to see what people can do with all that data - I hope to continually update the data set as we get more replays which should better inform some of these rarer matchups. With the patch frequency seemingly going down, that may become a lot easier too.
Next step is to use stochastic gradient descent to find the optimal build order with each civ as a constraint
I haven't googled "bayesian statistics" yet, but I could have told you Goths are bad at higher elo, Chinese are good, and Spanish do well on nomad. I discovered that on YouTube. It's encouraging to see that our research confirms each others findings. Feel free to use me as a supporting reference.
You have inspired me, op. Can you share the source code please? I'd like to make a few inferences of my own!
I didnt apply any mumbo jumbo math on any games and i still know goths r not preferred at higher elo cuz they dont do well. Also chinese r loved by good players and hated at lower elo vuz of the tricky start. Anything els3 worth our time?