Block 4: Errors and power
Week 12 · Uncertainty II · Imai §7.2.5–7.2.6 · about 4 min
NoteHow this works
Run each cell with Run Code, look at the output, then answer the question at the end of the block. Every cell is editable; Start Over resets it. Errors are information: read them.
Type I error = reject a true \(H_0\) (controlled by \(\alpha\)). Type II error = fail to reject a false \(H_0\). Power = 1 - P(Type II).
1. Power of a one-sample test, by simulation
A poll with n = 250; the true \(p = 0.48\) but \(H_0\): \(p = 0.5\). How often do we reject?
2. Two-sample power and sample size
Callback rate 5% vs. 10%.
Conclusion: What do the two results say about power?
The poll example has about 10% power (formula 9.7%): a 2-point deviation is almost never detected with 250 respondents, and failing to reject does not mean H0 is true. power.prop.test gives 0.85 for n = 500 per group and n = 581 for 90% power. A smaller difference (7% instead of 10%) needs about 2,960 per group.