It is not true that another finding in the same direction should always increase the statistical significance.

Suppose you've got a coin; null hypothesis is that it comes up heads and tails equally often. I flip the coin 5 times and get heads every time. Probability of at least this many heads on the null hypothesis is 1/32. Now I flip the coin another 9 times and get 5 heads / 4 tails: evidence in the same direction. Between the two experiments I have 10 heads / 4 tails. Probability of at least this many heads on the null hypothesis is [(14 choose 0) + ... + (14 choose 4)] / 2^14 ~= 0.09, much bigger than 1/32.

There are also other circumstances in which even additional strong evidence for X can reduce the probability of X.

Suppose I have three hypotheses A,B,C which initially I think are all equally likely. Then two things happen that both have probability 1/2 if A is true, probability 1/4 if B is true, and probability 0 if C is true. After one of them, I should think A is true with probability 2/3. After both, I should be more confident, right?

Nope. Suppose e.g. what I'm doing is pulling balls out of a bag. Hypothesis A is "I have either a bag of red balls or a bag of blue balls, with equal probability". Hypothesis B is "I have a bag with 25% red balls, 25% blue balls, and 50% green balls". Hypothesis C is "I have a bag containing only green balls".

So I pull out a ball from the bag and it's red. That happens half the time in scenario A, 1/4 the time in scenario B, and never in scenario C, like I claimed.

I put the ball back and shake things up so I'm starting afresh, and pull out another ball. This one's blue. Again: half the time in scenario A, 1/4 the time in scenario B, never in scenario C.

But those two things can't ever both happen in scenario A, because in that scenario I have a monochromatic bag. They can both happen in scenario B. And of course neither of them can happen in scenario C.

So I got a result that (on its own) was evidence for A over the other two hypotheses, and then another result that (on its own) was evidence for A over the other two hypotheses, and the effect of both together is that I know A is false and B is true.

Dynamical dark energy and the week that broke cosmology