We're going to begin this module talking about persuasion. And to do that we'll start with an interview that I did recently with a couple of colleagues of mine, Uri Simonsohn and Joe Simmons, about some research they've done recently. They published an article called False-Positive Psychology, Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Long title on a subject that seems a little arcane. And yet this paper has been wildly influential. It's been cited 2,000 times since it was published in 2011. It's arguably the most influential paper in what has been called the reproducibility crisis in psychology. And it raises a question, did persuasion have any role to play in the impact this paper's having? You might think academic research is a place where persuasion shouldn't matter. In fact, that's why I wanted to talk to Uri and Joe. Because you'd think this is a place where the only thing that matters is ideas. But it turns out these guys put a lot of thought into rhetoric and persuasion. So we sat down with them recently to find out how it went down. So we're here this afternoon to talk to Uri Simonsohn and Joe Simmons, both professors here at Wharton in the operations, information, and decisions group, about research that they've been doing for the last, what fellas, six or seven years now? It goes back a little ways. >> Six years. >> Six years. >> Six years on what they initially called false-positive psychology. And they published a paper in 2011 which, as of last count, has something like 2,000 citations in less than six years. And more papers behind that. And people generally give it considerable credit for the conversation in psychology around reproducibility. What some people will call the reproducibility crisis. And others have had voice in this, obviously, but the paper, and the research, in the ongoing program has been influential. We're here to talk today about persuasion, the role of persuasion and rhetoric in that effort. And the motivation is that we talk about persuasion all the time in other domains. We talk about it in marketing or politics inside a firm. But we don't usually think that something like science would involve persuasion and rhetoric. My sense of having been around you guys in this project for most of that time is that persuasion's been an important part of it. And so I thought it'd be good and instructive for us to talk a little bit about it. And hear kind of from the inside how you thought about that. And what your experience with persuasion and rhetoric has been in this domain. >> This was a paper that we really needed people to read and pay attention to. Otherwise, there's no point at all in doing it. We were trying change how the field as a whole did research. >> Okay. >> And so for the first time in my life I was involved in sort of a big persuasion attempt. And so we really did think hard about how on Earth to get people to read this paper. Most people don't read papers that are published. >> Even academics? >> Even academics, for sure, most papers do not get read. >> Right. >> So we're going to talk in a lot of detail about this research. It might be useful to explain in a very short fashion what the enterprise was. So what exactly were you trying to persuade people to do? >> [CROSSTALK] >> I know that's a ridiculous question, but it's concise. >> So basically change how people conduct and report the research. So the idea is people have, when you conduct the study, you collect many measures. For example, you may ask people how they feel about something. That you may ask them how hard they would work to obtain it. You may ask them multiple ways. But then, so if you collect all those measures, and you get to choose from them, the results you report are going to be biased towards the one that works. Imagine you try a drug, and you give it 100 patients. And you see if it helps with weight, or with their heart condition, or with their skin. And then if you try enough of those things, one of them will work. And then you report your results, and you only talk about the one that actually worked, readers will be misled into thinking that it's very persuasive, very compelling evidence. When in fact, it's just chance. >> Mm-hm, so this a important feature here, that this is basically a methodology paper. So it's not just persuasion and rhetoric in a scientific endeavor, but it's scientific methods. And so we're really getting into what I think most people would think is pretty dry, technical area. Not much room for persuasion, so how did you position this paper? How did you argue in this paper in a way that was different from other academic attempts? >> I mean, we were writing a methods paper. And most methods papers are written for methods people. And we didn't want to write a paper for methods people. We wanted to write a paper for normal people, normal academics, at least. >> Uh-huh. >> And we had the advantage of being normal academics. So a lot of people writing methods papers are methods people, but we are not a methods people. So we sort of understood, like what would it take for us to read a methods paper? And so we tried to build a methods paper that we ourselves would want to read. >> Mm-hm. >> So we started out by putting simulations in our paper because that sort of showed how big the problem was. But I think the big insight was that a paper that just contains simulations is not a paper that many people are going to read. >> Mm-hm. >> And so we thought long and hard, many, many conversations around what is a good example or demonstration of the problem? >> Mm-hm. >> And we tried to come up with a demonstration that was interesting, kind of cute, maybe people would talk about it, that sort of thing. >> I think that's sufficiently cute. You should tell us real quick, just to describe it at a high level. >> So basically wanted to show that scientists could find statistically significant evidence for any hypothesis, no matter how absurd. And so we came up with a hypothesis that was demonstrably absurd. Which is that listening to a particular song can change how old you are. And no one on Earth believes that's possible. >> Mm-hm. >> And so we ran a study using the norms of the day that basically showed that that effect was true. We were like, look, if you do things the way they are currently being done, you can find evidence for this crazy hypothesis. And you can write it up in such a way that scientists would be persuaded by it. >> Mm-hm. >> And so we thought that piece was, I think, absolutely critical. If we hadn't done that, I'm not sure we would've gotten published. But even if we had gotten published, it would've probably been ignored. >> Mm-hm. >> That was one aspect of it. >> So just on the persuasion, thinking about it, we would picture our least-mythologically oriented friend. And ask, what would get this guy to tell about this paper to his friends? What would the paper need to contain? >> Okay. >> So it becomes like viral for academic standards? >> Okay, where did that inclination come from? Is that a way you had thought about your work or your teaching? Why that pivot from the way you might typically write? >> The goal we said when we first met to discuss this, we said, okay. So we want to change how psychologists do science. And that felt like an impossible goal. And so the first step in persuasion, people have to listen to you. >> Mm-hm. >> And so I guess we were just very aware of how common it is for people to complain about a problem in academia and be ignored for decades and decades. And so the first step for not being ignored is people need to know what you said. I think, at least introspecting, that's what I think, [INAUDIBLE]. >> No, yeah, absolutely. So the first piece was basically getting people to want to read the paper by making it interesting. Maybe even a little funny in places, stuff like that. >> Mm-hm. >> But the second piece was about, as Uri was just saying, you can't just complain about a problem. You also have to propose a solution. Otherwise, people will just keep walking around complaining about it. And nothing will ever get done. And you also have to produce a solution that is actionable and easy and really hard to argue against. And so that was the other element of the paper that we really strove for. Other people at around that time or shortly thereafter were writing similar type pieces, but the solutions on offer were Utopian. In fact, some people even used the word Utopian. And they were not at all practical. And they weren´t going to happen any time soon, and probably not in our lifetime and probably not ever. And so we were trying to think instead of what could journals, what could scientists do tomorrow to change how they're doing it? And so we really tried to find a baby step that would actually work, actually be impactful, and that was sort of hard to argue with. So those are sort of the two elements we've really focused on, almost all of our conversations. I mean, the science of it was figured out in a day or two or something. Almost all of our conversations with this paper were about how to persuade. >> Wow. And the paper itself is actually quite short. It's like nine pages and two or three of those are these prescriptions. Long list of prescriptions. >> Right. >> But obviously the persuasive effort goes beyond the publication. You then go out and, or even before the complication, you're out talking about the paper, you're visiting schools, you're giving talks. How did you find yourself going about that differently than you had before? >> I think, that we did not alter very much. We did what we always did except that this time it was an incredible amount of demand to hear us talk about it. So we submitted a symposium to a conference which is something everybody does all the time. You say here's three papers, please let us talk about it and they said okay. And usually 20 people show up, 50, maybe 100 will show up. We had like 800 people. >> [LAUGH] >> It's like standing room only and that was, it's not like we prepped for that. It just happened. And I think it was really just the downstream consequences of having drafted the message in the paper appropriately. >> But that must have been reinforcing in some way. You had this pivot to thinking in a more, thinking about rhetoric more explicitly. I'd be surprised if you didn't bake that into some of your presentations. Did you not spend extra time, more than usual working on just the right example or just the right analogy? I feel like you guys are always coming up with these analogies that are just so apt. >> Yeah, I mean I practiced like crazy for that talk, for sure, to try to make sure all the words were right. But I think that was less of a concern about coming up with the right example in the talk. because basically we're using the example from the paper. >> Okay. >> So that wasn't really the issue. It was more about in my class on the decision making, I talk about motivated reasoning and how people are more likely to believe something when they want to believe it. And a big implication of that is they don´t, they´re more likely to believe you if they like you than if they dislike you. If we come out at the symposia and start talking about, hey, you guys are all doing this wrong, and we´re better than you, and we come out swinging like that, that's terrible, and false. because we had been doing all of the things that we pointed out in the paper. You can look at some of the garbage that I published prior to 2011, it's terrible stuff. And so we were also very cognizant of, the language was about we as a field. And in the paper, and in talks, we'd emphasize how we had all done these things. And now all of us, including us, now need to change the way we're doing things because you don't want to get on your high horse. People are going to hate that. >> Beyond just the first paper, you're also very well known for your work on p-Curves, and in that paper you came up with this term p-Hacking. You've coined a few phrases now that people know well. False Positive Psychology was the title of the first paper. This term, Researcher's Degrees of Freedom. What role do you think getting the right terms or titles play? You'd like to think as academics, come on, this shouldn't matter. This is superficial. But it kind of feels like it mattered here. >> I feel like it matters. It allows conversations to more Naturally flow. I've had a couple of experiences where I've really struggled, in a slightly different stream of research, really struggled putting forward the ideas in a way that resonates with people. And then I decided, what if I just call it something instead of saying a new approach to evaluating replications, and I played with ten different iterations of those words in a different order. And then I said, actually, this is really analogous to having a small telescope. So I re-branded the paper The Small Telescope's Problem and it immediately became a much easier idea to convey. When you want to talk about that paper, you said The Small Telescope Paper. You don't have to say the paper where Simon said that this thing and it becomes like a two sentence thing. >> So, I get that entirely, but I also have been aggravated for years. We all have been by researchers who are forever naming things. Kind of unnecessarily, it seems. Even well known researchers sometimes, forever want to came up with new heuristics, or a new something. >> Right. >> How would you characterize a successful, effective attempt at naming something new, versus kind of a superfluous, which is something we see. So it's just not go give something a new name. You have to do it in some thoughtful way. >> So at p-Hacking for example, the first paper, the [INAUDIBLE] one used researches versus freedom. And we used that I don't know let's say ten times. But then for the following paper, we had to use that every other paragraph. >> And it just didn't seem sustainable to use research the use of freedom 100 times. And we don't like acronyms. We didn't want to have RDFs everywhere. >> Okay, okay. >> And so we said we need a word. And so it was really, there's a concept in need of a label. Let's find a label, as opposed to, let's create an artificial need for a label so that we can become famous or something. >> Okay, okay. >> So I think that- >> Which is the way it feels sometimes. >> Yeah, sometimes it feels like, well, this is really similar to it but if I call it different, maybe we will perceive it as different. That's not, I don't think I would advocate for that. >> Absolutely not. >> But if you have a concept that you have to repeat and every time you repeat it, you have to define it, that basically the term is a definition of the term. That's when you need a word and a term. >> What missteps have you made? You guys have had, and you're part of a broader effort, of course, big impact. So, you've done lots right. Do you think there are any places that you've missed, or that you would of done differently? >> Yes. [LAUGH] Absolutely. >> Such as? >> So early on, we would get, because we had zero experience on this, right? So we had seven years of experience as researchers but zero as persuaders, right? And so, when people would push back and say, engage in personal attacks. Questioning our integrity. Or just any criticism, even beyond the criticism of integrity. Just any criticism of what we were saying, the facts, in a very public arena. It felt like an imperative to immediately and forcefully respond very emotionally. Just because it felt wrong to do it. >> Right. >> And we did, at least I would not do that anymore. So first of all, it is okay for people to have wrong beliefs that they express publicly about your work. You don't have to respond to all of them. And if you do respond, it's fine for you to remain cool. There's almost no downside to taking the higher moral ground. And we were talking about this intuition, that we can't remember where we got it from. Where the better it feels to say something, probably the worse it is for you to say it. >> That's nice. >> [LAUGH] >> It's a great heuristic. >> And unfortunately, I don't know where it comes from. So if we're writing something, if it feels great to say it, we try to erase it. >> Mm-hm. You've written, I think it was you writing, it might have been a more general effort, on essentially argumentation in science. Because you guys have been in this debate, you've kind of learned. Some does and don'ts. >> Right >> What else would you say is important? >> Yeah, that's a mansplaining. >> That's right >> [LAUGH] >> That didn't catch on. Not yet, at least. So I think there are three principles. I'll see if I can remember them. One was to not, to not seemed know the intentions of your counterpart. It is very common for people who are criticized to think they know the intentions of the other part, and that those intentions are non benign. So it´s a personal attack, for jealousy or for political reasons. And the point is, it´s not like there´s never, of course there´s attacks that are personal. But you don´t know, as a recipient, why it comes. And it's irrelevant, the criticism is either right or wrong. And if is wrong but for the right reasons it doesn't matter, it's still wrong. Don't pay attention to that. The other one was like reaching out, if you're going to discuss somebody else's work, send them a draft and ask them what they think. It's funny, I've talked to some journalist friends about this, this would be impossible, it's immoral. For you to share your story with the person you're describing before you publish it. But we have a blog and we do it. It's amazing. >> Listen, to be clear, it's immoral because of the rules of journalism is this objectivity. >> That's right, they feel like they're be compromised. I don't really buy the argument, because it's not like you tell them, okay, you write it up and I'll publish what you send me. You set out feedback. So often people get upset with journalists, or with bloggers, or with scientists, because they feel they've been mischaracterized. Almost every time we share the draft of a blog post where we discuss somebody else's work with them, they get incredibly upset about the tiniest thing that you couldn't care less about changing. And you change it, and they feel like you've ceded a lot of ground, and you feel like the piece is indifferentiable from what it was. >> Right, so by the way, that seems to connect to that quote about you rarely change someone else's mind without changing your own. So it's a form of negotiation almost you get just a little bit, but it feels great to them. And it helps in the persuasive effort, essentially, if you can give a little bit. And there was a third principle in? >> Yeah, the third is describe on label. So don't, so for example, if you're going to say something is, if you feel something is wasteful and you call it wasteful, all you're really saying is I don't like it. You know, really helping the recipient of the message to understand why you find it wasteful. And once you've explained why you find it wasteful, you don't really need the labeling anymore because anybody who reads the description will incorporate that. So, you almost never need to tell the recipient what you think. It's interesting why you think, and they'll figure it out. From my experience, the mixed message is much more difficult to counter. Especially, because terms can be so loaded and so easy to be interpreted in the wrong way. Especially, if you're the person who is being thrown the turmat. >> And Yuri, you just apologized for doing exactly this thing. You wrote this piece and then within the year you violated your own principle >> Couple of times, yeah. >> But then you also kind of confessed to it straight away and say this is >> Well, you know, it's very natural to, to feel good, to want to feel good with what things you say, right? It's like, you know when you throw the punch, that second feels great. >> [LAUGH] >> Right, it's the rest of the life you regret it, but that instance it feels great. >> Right, right, right. All right, anything else, fellows? Anything else about rhetoric and persuasion you think we haven't touched on that's especially important? Or that you've learned along the way? >> And just related to what Yuri was just saying is I think he's a million times better than I am at all this stuff he was just suggesting. And I think we've also really benefited from being a team of three. So Leif Nelson is also part of our team. And he's at University of Berkley. And a team of three is sort of, for us I think has been critical. Because there are times when two of us get very worked up and emotional, and want to respond to an email in a certain way. Or I want to post something online in a certain way, that violates these kinds of principles. And there's almost at least one of us who has the cool head. And we always check with the other two before doing anything. And so it's especially for me, it's really been helpful over the years because I could of, if I was doing this on my own because I'm a more emotional person, and basically I'm a bigger idiot is what I'm saying. I would have probably done a lot of damage to the cause. >> And do you eventually, start internalizing some of those voices and perspectives, having worked in that group? And Leif, great, let's recognize Leif as well, but having worked Leif and Yuri as close as you have for these years at some point, you'll never not need of it all. But do you need them a little bit less. >> I mean, yesterday we wrote something. We sent it to Leif, isn't this great? And he was like, no, you really don't want to send that. And so we turned like this page long email into a very polite two sentence. >> Yeah, yeah. >> And so, it's hard, because when you're experiencing the frustration or the anger, it's just really hard to. >> Yeah, the best you can do is have policies. I write emails and then I don't put the person's name in it who I'm directing it to so I don't accidentally hit send. And then after I write it, I'm like, that's going to feel really good to send. And then I save it or I forward it to them and ask if I can send it. And 100% of the time, they would say no. >> [LAUGH] >> You cannot send it. Or I would not send it to them and realize two days later that, of course you can't send that. You need to sort of calm down. But these are policies. My experience with this stuff unfortunately, has not changed. So the misery is still there. >> [LAUGH] >> Of dealing with this kind of stuff. >> That's profound. >> Yeah, it's not that different from overbuying or overeating when you're hungry. It's an emotional drive. >> It's that visceral? >> Yeah, it's a visceral reaction. If it's very basic instinct of expressing your anger when you experience it, so. >> Yeah, I sort of feel bad. There's lots of people in the field who've weighed in on these arguments and they've made, some people have made just massive missteps by. Posting things online and I feel they're going at it alone, so they don't have the person whispering in their ear or chaining them to the floor. >> [LAUGH] >> You know, telling them not to do this. >> Right. >> And so we've had the benefit of at least having the three of us. And I don't want to say that I think there have been times where even all three of us, sort of, together did the wrong thing but. But it's way, way less frequent than it otherwise would have been if we'd gone it alone. >> Great. All right, guys thanks for the time, very much appreciated. >> Yeah, thanks again, appreciate it.
