Introduction to EBPM (Evidence-Based Policy Making) Episode 4: Design Matters More Than Analysis (Supplementary Notes on Impact Evaluation) (Japanese)

In understanding impact evaluation, several important cautions should be noted: (1) statistical significance, a concept frequently used in evaluation, should not be overemphasized; (2) results that once showed a significant effect may fail to replicate; and (3) the presence or magnitude of effects may differ across units (e.g., individuals, firms). Although methods such as instrumental variable estimations (IV) and propensity score matching (PSM) are frequently used for impact evaluation, there are cases where it is difficult to determine whether the results are reliable. Practitioners in policy settings are therefore advised to approach these methods with appropriate caution. To conduct effective impact evaluation, preparation at the policy design stage is crucial—indeed, design matters more than analysis. It is desirable for those involved in policy design to consider appropriate strategies such as creating groups that receive interventions earlier than others based on random choice, and avoiding institutional designs that simply include all applicants as beneficiaries of the programs.