Sunday , February 25 2018
Home / Impact Evaluations / Your go-to regression specification is biased: here’s the simple way to fix it

Your go-to regression specification is biased: here’s the simple way to fix it

Summary:
Today, I am writing about something many of you already know. You’ve probably been hearing about it for 5-10 years. But, you still ignore it. Well, now that the evidence against it has mounted enough and the fix is simple enough, I am here to urge you to tweak your regression specifications in your program ...

Topics:
Berk Ozler considers the following as important:

This could be interesting, too:

David McKenzie writes Weekly links February 23: tell better stories, hot days = lower profits, women need more customers, and more…

Tony Yates writes MPC not as split over £ as they made themselves out to be

Reserve Bank Australia writes Payments System Board Update: February 2018 Meeting

IMFBlog writes A Digital-Savvy Indonesia

Today, I am writing about something many of you already know. You’ve probably been hearing about it for 5-10 years. But, you still ignore it. Well, now that the evidence against it has mounted enough and the fix is simple enough, I am here to urge you to tweak your regression specifications in your program evaluations.

A new paper by Gibbons, Serrato, and Urbancic in the Journal of Econometric Methods (a) shows that OLS with Fixed Effects is not a consistent estimator in the presence of heterogeneous effects; (b) develops estimators to deal with the problem; and (c) shows that the bias can matter substantively in some cases. You can download commands for R and Stata here if you wish. The problem in a nutshell is that the FE specifications are not fully flexible – allowing the intercepts of the conditional expectations to vary but not the slopes.

The implication, however, is simple and we have covered it here at least twice in the past five years. When you have covariate-adjusted regressions, including your block fixed effects, etc., demean (or center) your covariates and fully interact them with your treatment indicator. Then, the coefficient on T is still the average treatment effect, which is now unbiased and consistent even in the presence of heterogeneity (along these covariates). Winston Lin wrote about this in a guest post almost six years and David blogged about this when he discussed Imbens and Rubin’s book two years ago (see point 4 here). Both Lin (2013) and Imbens & Rubin (2015) prefer this specification for covariate-adjusted regressions. If you did not believe them or you thought this is not important enough, then maybe this paper will put you over the edge…

A couple of caveats: First, this will be less useful when you have a ton of fixed effects. For example, you may have pair-randomized the treatment assignments and won’t be able to establish heterogeneity in treatment effects from pairs. The advice on this usually is that you should have blocks (strata) for random assignment that are small, but not pairs. With lots of FE, you may still run into degrees of freedom issues with small sample sizes, however.

Second, Gibbons, Serrato, and Urbancic emphasize the fact that the adjustments make a difference. However, a quick look in their Appendix D.3 Table 6 suggests that in only one of the eight papers they examined, the correct specification makes a big difference. In all the others, the results are what the original authors would have called “robust” to different ways of handling this issue. My personal experience is also that this has not made a difference to findings in my own experiments thus far. The Karlan and Zinman (2008) paper is a perfect example for them where the variance in Treatment is much larger in one sub-group AND in the presence of effect heterogeneity along that dimension.

So, there you have it. Just add a few lines to your code to center covariates and create interactions and you’re set. No reason not to do it…

Berk Ozler
Berk Özler is a senior economist in the Development Research Group, Poverty Cluster. He received his B.Sc. in Mathematics from Bosphorous University in 1991, and his Ph.D in Economics from Cornell University in 2001. After working on poverty and inequality measurement, poverty mapping, and the 2006 Word Development Report on Equity and Development earlier, he decided to combine his interests in cash transfer programs and HIV risks facing young women in Africa by designing a field experiment in Malawi. He has since been involved in a number of cluster-randomized field experiments.

Leave a Reply

Your email address will not be published. Required fields are marked *