Hello and welcome back, today I'm going to talk about Controlled field experiment. In previous videos we have already discussed eye-tracking, field studies, alpha release, A/B testing as evaluation methods. In the next video we will talk about log analysis. In this video we will focus on controlled field experiment. Before I go into the details of controlled field experiment, I want to first remind you the goals of evaluation. The goals of all these evaluation methods is to quantify the effectiveness of different designs and inform future designs. Because these evaluation methods are often expensive to conduct, we yearly perform evaluation at key point of the design and at the end. Controlled field experiment is to experimentally examine different designs in the real world. To conduct a controlled field experiment we need to randomize subjects, that is users, into different groups using different designs and compare outcomes between these groups. Actually, A/B testing is a form of field experiment with only two variants, A and B. In this example I'm going to or in this video I'm going to show an example. In this example we used a field experiment to examine the effects of different types of feedback on Wikipedia editors' contributions. We already know that feedback is a very effecting mechanism to encourage the contributions in Wikipedia. In this specific experiment we want to compares the effect on this a four different types of feedback, Positive feedback, Negative feedback, Directive feedback and Social feedback. Here is an example of a positive feed back, this is a real message sent to Wikipedia editor. I am so impressed, this is a very fine article. Here is an example of a negative feed back. There is a concern the rationale you have provided for using this image under fair use may be invalid If it is determined that the image does not qualify on the fair use, it will be deleted within a couple of days. This is an example of directive message. Hey, one of these days do you think you could take some pictures at Mission Mill? I'd like to spruce up the article but it really needs some photos. Thanks! This is an example of a social feedback. Drop me a line on my talk page sometime, we will get a coffee over some place. In this experiment, we use 2 x 2 x 2 x 2 factorial design. Here is a definition of a factorial design. A full factorial design is an experiment whose design consists of two or more factors, each with discrete possible values or levels. And whose experimental units take on all possible combinations of these levels across all such factors. In our example, we have four factors which are positive feedback, negative feedback, directive feedback and social feedback. For each factor there are two levels, with this type of feedback, and the result of this type of feedback. Our experiment design takes on all the possible combinations of these levels across all four factors, which result in 16 different conditions. We randomly assign our participant into these 16 conditions. Particularly, our subjects who are Wikipedia editors who just created new article would receive messages from us. The messages have different combinations of the positive, negative, directive, and social components based on which conditions they are in. So first, all the messages will contain a base message like this. I just thought I'd let you know that I saw your the article in New Article list. For the participants who are in the conditioned with positive feedback they will receive a component like this. The information is presented clearly and is easy to understand. Similarly, for those participants who are in the negative feedback conditions, they will receive a component like this. However, I noticed the article contained an error. This article currently does not contain any references. As a new article the most important thing is to find reliable references for all existing information. For the participants in the condition with the direct messages they will see a component like, it would be great if you would also upload a picture for the related article. Similarly, for the people in the social feedback conditions, they would also have a social greeting and a social closing. Now let's see the results of this experiment. We will show you the main effects of the different types of feedback. First is this graph I will show you is the newcomers' edits on a particular article which they receive feedback on. Here, we will show the number of edits changed in the subsequent three months after getting the feedback message. In this figure, the wide bars indicate those participants who do not received that particular feedback component. While the color of the bar indicates those people who receives that particular type of feedback. So this figure shows us that the negative feedback and directive feedback actually significantly increase the people's edits on the article. However positive feedback and social feedback does not significantly increase people's edits on a particular edit where they receive feedback on. The second figure we are going to show how these newcomer's overall contribution changes after they receive our feedback message. Here, we see a opposite pattern. Positive feedback and social feedback actually significantly increase people's overall contribution. While negative feedback and directive feedback does not change people's overall contributions. Regarding the reasons behind these results, please refer to the paper which I put other resources of this video. Takeaways, field controlled experiment is a evaluation method that is to experimentally examine the different design in the real world. A full factorial design is an experiment whose design consists of two or more factors. Which was discrete possible levels, and whose experimental units take on all possible combinations of all these levels across all such factors. Thank you for watching this video. Hope to see you in the next one.
