Block 1: The logic of a test: the lady tasting tea

Week 12 · Uncertainty II · Imai §7.2.1 · about 4 min

NoteHow this works

Run each cell with Run Code, look at the output, then answer the question at the end of the block. Every cell is editable; Start Over resets it. Errors are information: read them.

The lady tasting tea: she claims she can tell whether milk or tea was poured first. 8 cups, 4 of each kind, in random order; she classifies all 8. Null hypothesis \(H_0\): she is guessing.

1. The truth by counting

Under \(H_0\), count the assignments that give 0, 2, 4, 6 or 8 correct cups (70 in total).

2. Fisher’s exact test

Conclusion: Which result gives a one-sided p-value below 0.05: "all 8 correct" or "6 correct"?
There are 70 equally likely assignments. Exactly 1 gives 8 correct (1/70 = 1.4%); P(6 or more) = 17/70 = 24%: a guesser does that one time in four. 8 correct is strong evidence, 6 correct is weak. The two-sided p-value for 6 correct is 0.49.