Comment: causal inference competitions: where should we aim?
Use soon-to-be-published RCTs to benchmark causal inference methods, similar to how computational biologists use soon-to-be-published solved protein structures to benchmark their protein folding algorithms.
Data competitions proved to be highly beneficial to the field of machine learning, and thus expected to provide similar advantages in the field of causal inference. As participants in the 2016 and 2017 Atlantic Causal Inference Conference (ACIC) data competitions and co-organizers of the 2018 competition, we discuss the strengths of simulation-based competitions and suggest potential extensions to address their limitations. These suggested augmentations aim at making the data generating processes more realistic and gradually increase in complexity, allowing thorough investigations of algorithms’ performance. We further outline a community-wide competition framework to evaluate an end-to-end causal inference pipeline, beginning with a causal question and a database, and ending with causal estimates.
Can causal inference methodologists use soon-to-be-published experiments (randomized control trials) to benchmark causal inference methods similar to how computational biologists use soon-to-be-published solved protein structures to benchmark their protein folding algorithms.
Citation
@article{karavani2019comment,
title={Comment: causal inference competitions: where should we aim?},
author={Karavani, Ehud and El-Hay, Tal and Shimoni, Yishai and Yanover, Chen},
year={2019}
}