In this video, I'd like to talk about Crowdsourcing. Crowdsourcing is a new form of organizing work that is mediated by Internet services. And it connects customers and workers through the power of the Internet. The prevalence of crowdsourcing is new. It's an idea that's been around for over a century. As James Surowiecki tells in his book the Wisdom of the Crowds in 1907, John Galton asked a large group of farmers to be able to estimate the weight of an ox. Now, what's interesting about this is that no individual farmer guessed the weight exactly. Yet amazingly, the average of all of the farmers' estimates was extremely close to the weight of that ox. You may have heard other similar examples of being able to guess the number of jelly beans in a jar or other kinds of crowd estimates of behavior sometimes markets and other systems can work in a similar way. What Surowiecki points out is that in order for the wisdom of the crowd to be able to work effectively, you need to be able to gather a large number of independent estimates. So part of the key of getting the farmers' votes or the jelly bean guessers is that each person guesses without revealing to anybody else what their guess is. And the reason for that is that in general, people's estimates for many tasks are uncorrelated. Some people underestimate and some people overestimate and that absence of correlation is what enables the averaging to be able to work. Another reason that this kind of averaging can fall short is if people tend to systematically make errors. And this is a subject that we're gonna return to towards the end of the video. So what we've seen with Crowdsourcing which is a term that comes from Jeff Howe's Wired article back in 2006, is that there's lots of things that distributing small tasks to groups of people who are each independently making estimates or coming up with ideas or doing other kinds of work. That this aggregate of groups of people can be really effective for things like, prediction markets on who will be the next president or who's gonna win the basketball tournament or things like that. It can be used though systems like InnoCentive to be able to solve difficult problem or being able to collect and filter information on the Internet. This is a lot of the user generated content that we see. And in many ways, this can be used a democratizing production enabling a larger number of people to participate in creating then otherwise would be possible. One of the most well known examples of Crowdsourcing is the Amazon platform called Mechanical Turk. And the reason that it's called the Mechanical Turk is that several hundred years ago, there was a traveling show of a supposedly automatic chess playing machine. And it was called the Mechanical Turk. It would on its own, play chess against the royalty in the places where it visited. The trick behind this automatic chess playing machine was that in fact, it wasn't a mechanical device at all under the hood, it was powered by people. And that's exactly what we're seeing with Amazon's Mechanical Turk. It's also powered by people. So there is an aggregate behavior and then underneath, there are people that make it all happen. The Mechanical Turk of old was one person underneath the table, hidden in a box. The new Mechanical Turk is aggregating the work of thousands of workers online. In many cases with this work you see things like, we'll pay a nickel to be able to find out whether these two images are of the same product. Amazon had many such cases like this, where their large warehouse of products means that there were many data integration and data cleaning and other kinds of organizing tasks that needed to happen. Much like they've done with other services such as Storage and Cloud Services. They realized that this labor market of being able to ask many, many simple questions with something that would be more broadly useful in the world. And the Mechanical Turk platform has been really widely used including by researchers. While Mechanical Turk is easily the most famous of the Crowdsourcing platforms, it's by no means the only one. And the Microtask, pay a nickel or a dime or even a dollar to get something done model that Mechanical Turk offers is not the only one. Another wonder example is oDesk, where in this case you're hiring a specific worker to do a small task. So the oDesk marketplace has people who profess skills at all sorts of things and then you can hire one of those individuals to be able to do some work. So at Mechanical Turk these are anonymous and in oDesk we know who they are and they have particular skills. But you don't need to use money to be able to make Crowdsourcing work, in fact I think some of the most interesting and creative uses of Crowdsourcing are where other incentive systems are in play. One that you've probably used and may not have even realized that it's a Crowdsourcing system is the reCAPTCHA system designed by Luis von Ahn and colleagues. And what reCAPTCHA does is the initial challenge was that Yahoo needed a way of preventing spammers from signing up for email accounts and other Yahoo based accounts through bots. So they needed some test to tell whether the entity signing up for an email account was a computer or whether it was a human. And so Louis and his colleagues creates CAPTCHAs or computer automated tests, to tell people and computers apart. And with these systems, Louis realized that initially, they would just distort text like you see on the screen here but there are many other things that you can use for the content of a CAPTCHA. So for example, Louis realized that you could use old New York Times articles that needed to be translated. You can feed a word or two at a time to the reCAPTCHA system. And the crowd when they sign up for accounts that need to verify that they're a human, we'll do the transcription work as part of that disclosure that they are human. So to prove I am human I tell you what the text says and then as a byproduct of that the New York Times gets translated, very cool. Now of course, you can't do this with just one worker you need to be able to ask multiple people about the same information. And you are going to need to compare the responses of multiple different individuals. More recently what we've seen are things like Google Street View has all of these street signs and house numbers and other information that they'd like to be able to extract and those are now handed off to reCAPTCHA. So the next time that you sign up for account, you may see a street sign or an old New York Time snippet or something else like that. These are in essence computer vision tasks that are just a little bit too hard for computers to do currently and we hand those to people. What's interesting is that then this massive store of human generated information becomes something that we can use to be able to train machine learning algorithms to do a better job in the future. And so there's a constant ratcheting that is happening, where the machines are getting better, by virtue of the insights being delivered from the crowd workers. So we've seen several different models of crowdsourcing. There are crowdsourcing systems that are extremely simple, very, very fast tasks. Others are much more creative. You can think about Kickstarter, and Wikipedia, and many of this other systems as being examples of more creatively oriented crowdwork. Now, also we've got things that are oriented toward novices. Anybody in the world can, in principle, tell us what the particular CAPTCHA says. Or for vision impaired users, we can offer an alternate modality with the Mechanical Turk tasks we're offering a nickel. So things like Amazon's Mechanical Turk and CAPTCHAS are very fast, very quick, easy. Broad labor pool available. Now other things may be more expert based. And they'd be more creative. And so for oDesk, for a graphic design task or something like that, we may have more complicated opportunities. And I think a lot of the interesting crowd work heads off in this direction of more creative expertise. And an awesome example of this is the work of Aaron Koblin and colleagues on the Johnny Cash Project. I believe this was Johnny Cash's very last music video. And what he did is that, they created an online painting interface, much like a Rotoscoping System. Where you could paint over individual frames of a music video for Johnny Cash's last song. Different users each contributed individual frames. You can see those along the bottom of the image here. And those were then composed together into a wonderful, crowd created music video. And so this was both a way to create a really cool visual effect. It was an outlet for creativity. And the fact that all of these different users were doing work that was showing up in one video together was a way to give tribute to the way that Johnny Cash touched so many people's lives. In fact, the very first concert I ever went to when I was five years old is, my parents brought me to an outdoor, I think free Johnny Cash show. And, it was amazing and so he's this enduring music legend and the crowd created video expresses that extremely well. As you might imagine, and I think what the Johnny Cash Project shows really well, is that people's motivations matter a lot. This paper by Chandler and Kapelner shows that when you give people a reason to do their work online. They do much better. And the result may not be totally shocking, but it's a really valuable reminder that when you situate the reason for why we're doing what we're doing, that can be extremely powerful. And the example that they show in this paper Is one of being able to find tumors in medical images. When crowd workers are told that they're simply locating objects of a certain style. The quality and time on task is much lower than when you tell people, hey, you're gonna find tumors and this is making a real difference. And then the workers do a much better job. And so for a variety of reasons whether it's a creative outlet, a tribute to an artist, the ability to potentially save or improve somebody's health. When workers are motivated and when the motivation is real, the quality of the work is much better. So what we saw with these simple tasks here like Mechanical Turk and CAPTCHAS, is there kind of one chunk of task all on their own? Or more complicated tasks. For example here's a paragraph. Can you shorten it or improve the grammar and writing? Well there's several operations that need to be done. And when online workers are asked to do more complex operations, there's a couple of challenges. As Michael Bernstein and colleagues point out, you've got the Eager Beaver, somebody who logs on online. They wanna show they know what they're doing. And so they'll make lots of edits. Or conversely, you can have the Lazy Turker. Somebody who wants their nickel for doing as little work as they possibly can. The way that you can combat both of these is to add a little bit of structure to the process. And one of my favorite examples of this kind of structure is Michael Bernstein and colleagues work on the find, fix, verify design pattern. And the idea here is that if you ask somebody to improve a paragraph, you may get junk, or you may get, well junk through over enthusiasm. But if you say, hey locate an area that could benefit from improvement. And then to the next worker you say, here's an area that could benefit from improvement. Could you improve it? And then to the third worker, you can ask, here is one version, here is another version, which two of these versions is better? By separating out the finding, the fixing and the verification, the quality of work that's done becomes higher. And this is a general purpose pattern that can be used for lots of crowd sourcing tasks. There are, of course, others. And so, what I mostly bring this up for is if you think about crowd creative content in your design work, one way that you can improve the quality of what you get back is by adding a little bit of structure. It's not the case that you can just magically throw any task out to the crowd, and get brilliant work back. It just doesn't work that way. In the Johnny Cash Project, we saw the structure being added through the rotoscoping tool itself, people didn't have open-ended painting, it was a very specific task. In Michael's work, with Find-Fix-Verify, the structure is being enforced procedurally. And both of those can improve the quality of the output. But we shouldn't assume that just because things are going out to the hoi polloi, that the quality of what comes back is no good or anything like that. In fact, as we see with the folded system, crowd workers can actually produce scientific insights that are novel in the entire world. And I love this research project because it illustrates I think a couple of important points. Foldit is a system where anybody around the world. Can use a game like user interface to be able to specify appropriate protein foldings. So in a way, you're asking crowd workers to do tasks that would normally be required of a person who's a PhD in the BioScience without that kind of training, some other training needs to happen instead. And so our first take away from the fold it system is that crowds can do extremely high quality work but you do need to train them up a little bit. And so with the folded game, there's a wonderful set of training and levels and other ways of on boarding new users to be able to learn the rules of the game quite literally. So you learn how proteins can and kind of unfold. Another thing that we see with folding is that some aspects of this tasks are both boring and automatable. And in those cases, we have an algorithm that does the local optimization of the folding structure. So users do the high level parts of the 3D task and then the computer helps and does the final polishing. And this is an example of what we in the design lab call, human and technology teamwork. That by using human intelligence and machine intelligence, smartly combined, you can end up with something that is of higher quality than either of the two alone. The last example that I wanna show today is the work of Walter Lasecki and colleagues on real time audio transcription. Currently to be able to have a class like the classes like I teach at UC San Diego, to have a class like that transcribed requires advanced notice and somebody whose trained as a stenographer. And the wages can be several hundred dollars an hour. And what that enables is that students who need the aid of captioning or transcription because of limited hearing can now participate in classes, and so this is great. However, the burden of being able to do this means that it can only be deployed in particularly high value situations. If audio transcription were easier and lower cost and more on demand, then we could open up more and more of the world to the hearing impaired. And this is what Walter and his colleagues worked on is by farming out the transcription task to the crowd, we're able to for much lower cost get something of comparable quality much more quickly and in an on-demand way. Now, I don't know about you but I can't type on my keyboard as fast as a court stenographer can. So we're not doing in this system is that scribe doesn't ask it's users to type the whole darn thing. You can't type for 50 minutes of a class and keep up. What Scribe does do, is that you get a little snippet that you transcribe and to make the hand off work between different people. So the first person does this snippet. The next person does this snippet. The next person does this snippet. There are several design cues that are necessary to make this work. For example, there's an audio queue where the volume gets louder when it's your turn to type. But you can hear the audio right before and right after. Also, speeding up and slowing down and several other things can improve the quality of the transcription. And this really underscores the fact that if you naively give a task to the crowd, you may get back total garbage. But that through very careful design like Walter has done here with Scribe and Michael did with Find-Fix-Verify and the other examples did in each of their domains, you can end up with something where the results are actually very impressively high quality. So I wanted to loop back to where we started, if we think back to John Galton's ox, the magic of the crowd generated gas was that the average produced something that was better than every single individual guesser. It doesn't always work that well but when the errors are uniformly distributed, the central tendency comes out and reveals the correct result. But let's think about this from an individual guessers perspective. We know that an individual guesser is likely to be biased. But it's hard to know a priori in which direction, so if I'd like to know how much the ox weighs or how many jelly beans there are in the jar, my guess is unlikely to be all that good. Ed Voolen is colleague here at UC San Diego, figured out something very clever. What they realized is that if you take an individual guesser and you say now, assume your guess is wrong in that case, what we can do is you can ask that guesser to make a second guess. And when that guesser guesses a second time for the reason of their guess being wrong. So if I were to have for example the jelly beans in the jar. I've got a jar, it's full of jelly beans. I might say, well I made my guess assuming there are ten jelly beans in each row along the jar. But what if it's not ten? What if it's 20? Well then my guess would double. And then you can ask me to do the same thing, say assume your second guess is wrong. What might be a reason for that and make a third guess. And when you average together all three of these guesses in many domains you can end up with a mean of those three covers about half of the distance between an individual guessers error. And the crowd in the middle. So if we have the right answer here and our crowd guesser guess all sorts of different stuff, an individual guesser's first guess maybe erroneous. Their second guess often balances that out and then we add the third guess in there, too. And the aggregate of these three is often quite good. And so what this tells me in aggregate, when we look at all of these effective systems for crowd sourcing together, is that design is critical to crowd sourcing success. And what I'd like to close with today, is the insight that we're seeing an increasing dominance of the Internet of things. That many of the services that are exciting right now are things that combine the world of atoms with the world of bits and the world of culture. This can be the pop-up food truck that announces where it's gonna be on Twitter. This can be Etsy, where we have people all around the world creating handy crafts and vintage items and then selling them through an online portal. This can be many common elements of political life that are coordinating large numbers of people through an online system. And so what we really see is that the Internet of things is really the Internet of social and technical elements working together. And so the ideas that you've learned in this crowdsourcing video are going to become increasingly applicable as the computational world and the everyday physical world merge more. So that's our video on crowdsourcing and we'll see you next time.
