Verifying the internal validity of a flagship RCT : a review of Crépon, Devoto, Duflo and Pariente : rebutting the rebuttal

We reply to CDDP's response to our replication of their published article in AEJ:AE. They reject most of the errors we documented in our replication paper. We provide a detailed answer to each objection they raise. We find that almost all of their rebuttals are driven by mistakes on their part. Once all these mistakes in CDDP's answer have been rectified, we find that all the coding, measurement and sampling errors documented in our replication still hold. All that remains then of the rejoinder is CDDP's argument that the issues we raised are not relevant because they do not substantially modify their impact estimates, and the use made by CDDP of additional sophisticated econometric tests to argue that their original results are robust. We disagree, as we find that correcting the rectifiable errors we identified does indeed show that the impact on assets and profits is not significant, and that the main results are to be found in increasing turnover from self-employment, which is trivial and generates very different conclusions to the original paper's findings. CDDP also omit to mention that the core conclusion of our replication was that, irrespective of the revised impact estimations, these results must be considered as lacking validity due to the massive inconsistencies found in the data, the substantial imbalances at baseline, the flaws in the experiment's integrity and the signs of probable contamination by other utility-related interventions. We are unable at this stage to assess the validity of the double post lasso procedure, the Benjamini-Hochberg False discovery rate correction of multiple testing, or the machine learning analysis put forward by CDDP, as they have not disclosed the related statistical scripts. Yet we fail to understand how even the most sophisticated methods could solve the "garbage in-garbage out" issue characteristic of this study. At this stage, we can only say that we have a very different notion of what underpins the internal validity of empirical research. We encourage CDDP to submit their answer to a peer-reviewed journal for a third-party appraisal of this debate.