The p-value, i.e., statistical significance is a cornerstone of evidence-based science including physics, chemistry, psychology, and sociology. It is equally important in understanding data analysis in corporations. Although, I would wager that most people don’t care, I would like to emphasize that one must understand what statistical significance is key to understanding whether your results are just a fluke. However, I personally feel like that the way most statisticians talk about it, it creates a lot of mystique and misunderstandings and only a bit of clarification. I have seen quite a few data-analysts who do not have a clear understanding of it even though they are able to produce the value through their software tools. However, a lack of understanding creates a space for misinterpretations, so I thought I would try to clarify it.
What exactly is a convincing evidence?
Well, let us say that Tom and Mary are playing a game of coin toss, they toss an unbiased coin and the loser pays the winner five dollars (or the loser takes a shot of tequila, take your pick). I agree that it is a very dumb and boring game, but bear with me. Mary notices that after playing three tosses, Tom has won twice. She suspects that the coin is biased in favor of Tom. Tom, of course, vehemently denies it and claims that he is just lucky. Does Mary have a case? Well, if you think about it, that is the most balanced the results could have been — after three tosses, one of the parties had to get at least two wins. Tom points this out and they continue playing. Now, after playing \(30\) times, Mary notices that Tom has won \(20\) times. Tom again denies such any wrongdoing. The question is, does Mary have enough evidence to claim that Tom is cheating? Well, it depends, after all, it is not impossible for Tom to win all the games even with a completely unbiased coin. And we should stick to the principle of ‘innocent until proven otherwise’ to any claims of cheating. But, that cannot work in practice because that way cheating will never be caught. So, we will have a probabilistic standard of evidence —
‘Innocent until the probability of such a lopsided result from a fair game is at most \(p\).’
Basically, we will consider the result to be too lopsided in favor of Tom if the probability of getting such an extreme result or even more extreme results is too little to ever actually occur in practice. The probability is considered too little if it is at most \(p\). This is also called the \(p\)-value, since statisticians are the most creative people when it comes to naming concepts. Of course, if the threshold \(p\) is too large, we run the risk of convicting an innocent Tom of cheating; and if it is too small, Mary cannot have much of a shot at ever convicting Tom. The correct choice of \(p\) depends on the context of what the claim is, most people in data-analysis, psychology, or sociology use the threshold of \(p\le 0.05\). Such a large value can have its own problems, but I will discuss that later. For now, let’s come back to Mary’s accusation against Tom. With the definition settled, all Mary has to show is that Tom’s wins are too lopsided. What is the probability of Tom getting \(20\) or more wins out of \(30\) tosses? The probability of Tom winning exactly \(r\) times is \(\binom{30}{r}/2^{30}\). So, the probability of getting the same or a more extreme result is —
\[ \begin{aligned} &\sum_{r=20}^{30} \frac{\binom{30}{r}}{2^{30}}\\ =& 0.049 \end{aligned} \]
This is less than \(0.05\), so Mary indeed does have a case and has successfully demonstrated that Tom is cheating, but Tom, of course, still denies it and claims that \(0.05\) is too high a threshold.
Let us now look at Mary’s argument a little closely. Mary’s claim is that Tom is cheating. This is what the statisticians would call the ‘alternate hypothesis’. The standard belief is Tom is innocent — this is what the statisticians call the ‘null hypothesis’. The argument is similar to proof-by-contradiction. In non-probabilistic logic, to prove that Tom is cheating, Mary has to prove that if we assume Tom to be innocent, the observed results would be impossible. In probabilistic logic, Mary instead only has to show that if we assume Tom to be innocent, the observed events would be extremely improbable (as opposed to impossible).
The universe of possibilities
In a different story, let’s say you live \(30\) kilometers (The Americans can just read it as \(30\) miles) away from your workplace and let’s imagine there is no public transport available from your home to your workplace, how do you get to work? If you ask Mr. Musk, he would of course tell you that your only option is to buy a Tesla. Whereas a Tesla can be a perfectly viable option, any reasonable person would understand that there are plenty of other options like cars from other brands and maybe even working from home.
Often times a similar situation arises when constructing a null hypothesis and an alternate hypothesis. Everyone knows that if you can reject the null hypothesis, you have to accept the alternate hypothesis. But this can be misleading. Let us take a linear regression problem for example. The data-analyst gives you a linear regression model from the observed data. The model is \(y=\beta x + \alpha\). The data-analyst also gives you the \(p\)-value of the model, say \(0.0000124\). Hence, the data-scientist strongly demonstrated that there exists a linear relationship between \(y\) and \(x\) with those specific \(\alpha\) and \(\beta\) (say \(\alpha_0,\beta_0\) respectively), right? Well, not exactly. First, you have to understand what the null hypothesis and the alternate hypothesis are. The null hypothesis, in case of a linear regression, is \(\beta=0\), i.e. the data are distributed with a normal distribution around a constant mean \(\alpha\) independent of \(x\). Since the \(p\)-value is so small, and you would accept any \(p\)-value less than \(0.05\), you can reject the null hypothesis. But what is the opposite of \(\beta=0\)? It is certainly not \(\beta=\beta_0\). It’s not even the fact that a linear model explains the relationship. The opposite of \(\beta=0\) is ‘either \(\beta \neq 0\) or the relationship between \(x\) and \(y\) is not linear, or the error is not normally distributed with a constant variance.’ When the data-analyst tells you that the linear-model exists with a very low \(p\)-value, they are implicitly assuming that if there is no relationship between \(x\) and \(y\), the \(y\) values must be randomly distributed with a normal distribution with a constant mean and variance, and when \(y\) dependent on \(x\), it is linearly dependent ; and even in that case, the only thing they are proving is that \(\beta \neq 0\).
In other words, the union of the null hypothesis and the alternate hypothesis is the universe of possibilities — i.e., the underlying unquestionable assumption. This is quite difficult to avoid though. Since you need to calculate the probability of the observed and more extreme result, your null hypothesis must make an assumption about the distribution of data points. That means, the opposite of the null hypothesis is every other probability distribution. That’s not an appealing alternate hypothesis to talk about. So, the underlying assumptions have to creep in some way or the other.
A related observation is that \(p\)-value calculation does not require too much from the alternate hypothesis, except the idea of which way the more extreme results are. Suppose in the game between Tom and Mary, the results are same (Tom won \(20\) of the \(30\) tosses), but instead of Mary, it is Tom that is accusing Mary of cheating. It would be a laughably weird claim to make of course, since Mary did not even do so well (talk about gaslighting), but how do we calculate the \(p\)-value in that case. Notice that the null hypothesis is still the same — no one is cheating. The only thing that changed is the alternate hypothesis. Now the more extreme side is Mary winning more. So, the \(p\)-value would be —
\[ \begin{aligned} &\sum_{r=0}^{20} \frac{\binom{30}{r}}{2^{30}}\\ =& 0.97861 \end{aligned} \]
Needless to say, this is way larger than \(0.05\) which matches with our intuition that it is a ridiculously weird claim coming from Tom. Our alternate hypothesis in this case is actually the logical negation of the null hypothesis which is every bias other than the fair probability value of \(0.5\) for each of them winning in each coin toss. In case of linear regression, the null hypothesis is \(\beta=0\), so a more extreme value is far away from \(0\), i.e., a high absolute value with the underlying assumption that the only possible relationship is linear. It is not even possible to specify what an extreme value without the assumption of linear regression since values of \(\beta\) would not even make any sense, the alternative hypothesis in this case would be too broad to nail down.
Probabilities are delicate
The concept of probabilities is inherently delicate, a tiny change in the description of the event can have a vastly different probability value. This means statistical significance of an experiment is also delicate. For example, again looking at Mary’s allegation, we have considered the case where Mary discovered the problem only at the end of their \(30\) toss streak. But we do not know what Mary was thinking of while playing the game after every toss. Suppose Mary was always waiting for chance to accuse Tom. After every toss, Mary calculates the \(p\)-value in the same method given above and accuses Tom whenever the computed \(p\)-value is smaller than \(0.05\). We see that this increases her chances of getting a successful conviction much higher. If the null hypothesis is still that Tom is innocent, but Mary is looking for a chance to accuse him, what is the \(p\)-value in this case? Of course, if the number of tosses is just a few, there is a good probability of every result. Let’s say Mary wins her case if \(p \le 0.05\). Computing the probability using a program, the probability of Mary getting a chance to convict an honest Tom turns out to be \(0.13\), which is quite a lot more than our allowed value of \(0.05\). In general, we need to be careful about what our null-hypothesis is and whether it actually corresponds to reality. We used the following code to compute it -
from math import comb, floor
def one_sided_pvalue(wins, tosses):
"""P(>= wins successes in `tosses` fair Bernoulli(1/2) trials)."""
return sum(comb(tosses, r) for r in range(wins, tosses + 1)) / (2 ** tosses)
def prob_accuse_innocent(N=30, alpha=0.05):
"""
Exact family-wise probability that a fair process triggers
p <= alpha at least once during tosses 1..N, under continuous peeking.
"""
# For each toss count t, find the smallest win-count that triggers accusation:
# threshold[t] = min { w : one_sided_pvalue(w, t) <= alpha }
# (If no w triggers, threshold is t+1, i.e. unreachable.)
threshold = [0] * (N + 1)
for t in range(1, N + 1):
thr = t + 1 # unreachable by default
for w in range(0, t + 1):
if one_sided_pvalue(w, t) <= alpha:
thr = w
break
threshold[t] = thr
# dist[w] = probability that Tom has exactly w wins so far AND
# has NOT yet been accused. Start before any toss: 0 wins, prob 1.
dist = {0: 1.0}
prob_accused = 0.0
for t in range(1, N + 1):
# advance one toss: each surviving state (w) splits into (w) and (w+1)
new_dist = {}
for w, p in dist.items():
new_dist[w] = new_dist.get(w, 0.0) + p * 0.5 # this toss: Tom loses
new_dist[w + 1] = new_dist.get(w + 1, 0.0) + p * 0.5 # this toss: Tom wins
# now apply the accusation rule for toss count t
thr = threshold[t]
survived = {}
for w, p in new_dist.items():
if w >= thr:
prob_accused += p # accusation triggered here; mass leaves the game
else:
survived[w] = p
dist = survived
return prob_accused, threshold
if __name__ == "__main__":
p, threshold = prob_accuse_innocent(N=30, alpha=0.05)
print(f"P(honest Tom accused at least once, peeking every toss, N=30, alpha=0.05) = {p:.4f}")
print()
# Show at which toss counts an accusation first becomes possible, and the win bar:
print("toss t : min wins to trigger p<=0.05 (and that exact p-value)")
for t in range(1, 31):
thr = threshold[t]
if thr <= t:
print(f" t={t:2d} : w>={thr:2d} (p={one_sided_pvalue(thr, t):.4f})")
# --- The other figure asserted in the article: the 'try 20 models' p-hacking number ---
print()
p_hack = 1 - 0.95 ** 20
print(f"P(at least one 'significant' result from 20 independent tries at alpha=0.05) = {p_hack:.4f}")Another issue with statistical significance is something called \(p\)-hacking; yes, it is so common that it has got its own name. To give you an idea, let’s assume that Tom and Mary play this tossing game every day and the coin is tossed exactly \(30\) times. Now, if Tom is honest and Mary is trying to find a reason to claim that Tom is cheating, how likely do you think that there would be a day when Tom would get at least \(20\) wins? If she keeps trying, this is bound to happen eventually. It is not uncommon for data analysts to inadvertently use a similar technique to find patterns that look statistically significant while there is actually no real pattern at all. If an analyst gets a dataset and tries 20 arbitrary models on a dataset that does not have any pattern, there is still a probability of \(1-0.95^{20}=0.64\) of getting a pattern that would be inside the \(0.05\) threshold for the \(p\)-value. To avoid this, there must be a test data that has been held out that can be used exactly once to check the model’s \(p\)-value. But this is almost never done because you can’t just throw away data after trying every model that you have generated.
What complicates the problem more is that researchers in a field study some effect all over the world and may do the same experiment (same experiments, but different research groups), some of them are bound to see an effect that is not really there with an acceptable \(p\)-value. This is why it is necessary to do systematic studies, where you count how many people got what kind of result, to make any real conclusions.
0 comments:
Post a Comment