In this session, we provide some examples of managerial applications of the difference-in-difference approach. Experiments have become increasingly useful tools for companies to address important questions, and make important managerial decisions. An ode by Oliver Hauser and Michael Luca on November 23rd, 2015, in the Harvard Business Review, highlights that companies are full of good experiments and only need to look for them. Moreover, today companies can run many digital experiments on the web, especially if they have web pages regularly accessed by customers. Nick Bloom of Stanford University and his colleagues conducted a famous field experiment on managerial practices. The article was published in 2013 in the Quarterly Journal of Economics. They studied whether the adoption of managerial practices, such as measurement of outputs, definition of targets, incentives, improve firm performance. The experiment first measured the performance of a sample of companies, such as productivity or profits, over several weeks before the treatment. It then offered free consulting services for five months to a random selection of these companies, while the control group only received one month. Finally, the experiment measured over several weeks after the treatment whether the treatment increased performance. They find that the difference in performance after minus before the treatment is higher than the same difference for the control group. The method is the one discussed in the previous session, with both individual and time fixed effects. Because of the randomization, the only thing that changed after versus before the treatment is the intervention. As a result, they attribute the change in performance to the adoption of managerial practices. To ensure that this is the effect, they also show that the difference in the share of managerial practices adopted after versus before the treatment by the treated companies is higher than for the control group. The result of this experiment has implications for actions. If companies with relatively poor managerial practices invest in them, they improve performance. Another example is an experiment we conducted at Bocconi University on the adoption of the scientific approach we have discussed in this course. We have taught a randomly selected group of 59 startups to adopt this approach, while to a controlled group of 57 startups we taught more standard managerial practices. This article is published in Management Science in 2019. Using the difference-in-difference approach, we find that the treated companies recognize to a greater extent whether the business idea is valuable and make fewer type I and type II errors. They are more likely to pursue ideas that become profitable, and they are less likely to pursue ideas that prove eventually to be unprofitable. Finally, consider the following practical example of a field experiment that you could run in a company: A supermarket chain is composed of 111 stores in four cities. The top managers want to assess the impact on the store's operating profits of two investments: one, a new marketing campaign; two, the introduction of a new digital technology. They first randomly allocate stores in three groups of 37 stores. One group receives no treatment, a second group runs the marketing campaign, the third group introduces the new technology. The data of these experiments are in the dataset promotion versus technology.xlsx. There are 222 observations, 111 stores times 2 periods, before and after the treatment. In the dataset, you find the index id for the 111 stores, the dummy period equal to 0 before the treatment, and 1 after the treatment, two dummy variables, T1 and T2 for, respectively, the marketing campaign and the technology that take values 1 for the treated groups during the treatment. A variable city takes the value 1 to 4 according to the location of the store, and finally two variables, neighb_pop and employees, account for the neighborhood population and the number of employees of the store. After importing the data in Stata, you xtset id and period. You then run the xtreg regression with dependent variable operating profits, and all the other variables as independent variables in the dataset. To obtain the city dummies you employ i.city like in the previous session. You will estimate only three city coefficients, as the fourth one is embedded in the constant term. Before we discuss the results, please note that we have made up these data for illustration purposes, the results are not representative of real situations. With this caveat in mind, the difference-in-difference estimators are the coefficients of T1 for the marketing campaign, and T2 for the technology. The coefficient of the promotion is small with a very high p-value. This implies that with the 95 percent probability it could be generated with the null hypothesis that the true coefficient is equal to 0. The 95 percent confidence interval is also centered around 0. This strongly suggests that the promotion has no impact on the operating profits of the stores. Conversely, the coefficient of technology is sizable and statistically significant, as implied by the p-value and the confidence interval. This suggests that the technology does make a difference. Unlike the correlations produced by simple regression analysis, the size of these coefficients are meaningful, and they indicate specific effect sizes. In particular, the T2 coefficient predicts that - other things being equal - a store that adopts the technology will enjoy an increasing operating profits of nearly €170,000. The results of this analysis then suggest that it is worth investing in the new technology to increase operating profits, while it is not worth investing in a new campaign. This analysis has implications for action. A few other issues are worth noting about this analysis. First, if you run the xtreg with individual fixed effects, you will not be able to estimate the coefficients of the city dummies. This is because the city location varies across stores but not over the two periods. You will instead estimate the impacts of the neighborhood population and employees, because they vary both across stores and over time. When you employ individual fixed effects, you cannot employ any independent variable that varies only across individuals. The individual fixed effects capture any difference across individuals, but the downside is that you cannot estimate the effect of the variables that vary only across individuals. In regressions like these, the focus of your analysis is the causal effect produced by the treatment, and therefore, you'll probably only use individual fixed effects. However, if for any reasons you are interested in the effect of any other variable, that only varies across individual observations, you cannot use the individual fixed effects. Second, you can verify by running the regressions that whether you use the city dummies or the individual fixed effects the treatment coefficients do not vary much. In contrast, if you do not include period in the xt regression, you will find that the impact of the treatments increased considerably, and the impact of the campaign becomes sizable and statistically significant. This reinforces the argument discussed in the previous session. In difference-in-difference regressions, the time is correlated with the treatment. It is, therefore, crucial to control for it with time dummies, otherwise the treatment is likely to pick the effect of time, and it will be biasing the estimation of the causal effect you are interested in.
