Thursday, July 19, 2012

Why I don't think publishing more negative results is worth any effort at all

This is part II of my reaction to the standard litany of well-intentioned ideas to improve science.

One general refrain I hear everywhere is the need to avoid the file drawer effect, publication bias, and more generally the large dark matter of unpublished results floating in the center of the universe that, were it published, would save everyone lots of time.

While I am equally opposed to the file drawer effect, publication bias, and the agony of re-following someone else's doomed footsteps, I dont think the solution is as easy as publishing negative results. And definitely not requiring publication of negative results.

[edit: Obviously many negative results are published currently and Im not opposed to that. Im talking about requiring publication of all negative results.]

Back in my undergraduate days, in physical chemistry, we learned the reason for the 2nd Law of Thermodynamics. There is only one way to be ordered and many many ways to be disordered.

The same principle applies to science. There is only one reason your experiment worked: you did it right and your hypothesis was right*. There are many many reasons it didn't work: most of them are the huge number of ways you did the experiment wrong. One of them is that you did everything perfectly but it just happened your hypothesis was wrong. But that's just one.

[* It's also possible it worked by chance, but we already deal with that possibility through hypothesis testing.]

Im not just talking about making basic mistakes. There are many levels at which problems could creep in. I won't enumerate them. Suffice it to say, science is hard. Even those of us who have been doing it for 12 years (=me) still screw up more often than we get it right. I have oodles of negative results in my file drawer (technically in folders on my laptop). Some of them are probably negative results because I did silly stupid sloppy things I would be embarrassed for anyone to know about.

I suppose I COULD take the time to do everything right, but maybe I used a task that is slightly wrong. So I should try all slightly different tasks. Or maybe I was recording a few millimeters away from the hotspot. So I could go explore all the adjacent areas. I COULD do that, but I don't have all the time in the universe. And until I do, my negative results are not interpretable by anyone.

We have a current system for publication of negative results. It's called word of mouth. I recently got interested in Exciting New Brain Area "X". I casually asked a couple people what they had heard about it. They told me they heard persons Y and Z were recording there. I casually asked Y and Z for their reactions. They both told me they got negative results. Their experiences gave me some guidance on how to proceed. They did not have to spend a month preparing manuscripts summarizing their findings. They just had to talk to me for a minute or two. Much less effort for them!

This is why I am all in favor of grad students travelling more and talking to other scientists. I think that informal communication conveys as much information as published papers and talks. But that's a topic for another post.



5 comments:

  1. We have a current system for publication of negative results. It's called word of mouth.
    But now you assume that everybody knows everybody, which is true in a lot of (American) science-land, but I know that when I started my PhD, my PI had just started working on the subject that my PhD was on, and we didn't know everyone in the field (with that I mean: we almost knew no-one in the field). And in cases like that, it is really nice if there's a journal or database where you can publish your negative results (or even better: things that just don't work at all, meaning that they don't even give you data for some reason).

    ReplyDelete
    Replies
    1. I totally agree with you. And it's even worse for Australian scientists I have spoken with.

      The problem is, there's plenty of motivation on the demand side, but there is very little motivation on the supply side. How can you incentivize those people who know what doesnt work to tell the world, without requiring them to go through the rigamarole of publication?

      Delete
  2. Would blogs have a role here/

    ReplyDelete
    Replies
    1. It would be great if public forums existed where people could chatter about this stuff.

      I think many people are overly worried about getting scooped, but those people might not talk to you in person either.

      I dont know if that exists for my field. Right now I just email relevant people, but only occasionally.

      Delete
  3. Nice post. You put in words some of the things I've been thinking as this debate continues. Your point about entropy (many ways to be disordered) is a good one. Null results don't tell us a lot - though they add power.

    ReplyDelete

Note: Only a member of this blog may post a comment.