I totally agree @mrfuji3 - I was just a bit tired of testing/writing down results manually ;-) .
I thought the data might be large enough to encourage someone at Larian to look into it with either better tools (or just look at the code directly) instead of spending more hours, but I also had some doubts which tempted me to collect more data, to reduce the uncertainty.
That's fair. Kudos to recording all those results and doing this analysis!
I've done a bit more analysis using your raw data.
Combining all the Karmic Die Rolls (735 total rolls)- The average is 12.33 +/- 0.22, inconsistent with the expected average of 10.5 at >3-sigma.
- 18s, 19s, and 20s appear too often (1.5, 1.75, and 2x more frequent than expected for 3, 4.7, and 6.3-sigma differences, respectively.)
- 2s through 9s don't appear often enough (roughly 60-70% as frequent as expected, for ~2-sigma differences).
- Somehow, natural 1s occur ~exactly as often as expected, 1/20th of the time. Perhaps Karmic Dice doesn't re-roll 1s...? This would tie in to the larger issue in BG3 where natural 1's override everything.
- Natural 20s occur about as often as they would in an Advantage system (~10%), but this is not true for other values.
- The full distribution of rolls is inconsistent with a uniform distribution, with a chi^2 value of 111. WAY larger than the critical value of 30.144 for 19 degrees of freedom
- Individually, we also reject the hypothesis that each "vs X AC" set of rolls is consistent with uniform, as each has a chi^2 value greater than 30.144.
Combining all the non-Karmic Die Rolls (634 total rolls)- The average is 10.42 +/- 0.23, consistent with the expected average of 10.5.
- All values appear within ~25% (<1.4-sigma) of their expected frequency, with most falling within 15% (<0.8-sigma).
- The full distribution of rolls is consistent with a uniform distribution, with a chi^2 value of 12.18. Way smaller than the critical value of 30.144 for 19 degrees of freedom
- Individually, we fail to reject each "vs X AC" distribution as being distinct from a uniform distribution.
- The higher frequency of natural 20s in the "vs 23 AC" data might just be within expected variation (less than 2-sigma difference; e.g., within the 95% confidence interval).
