I agree that there are problems in science, although I think (following research by Dan Ariely) that major cheaters are relatively rare and that minor cheating is near universal. I also doubt that cheating has increased, but I have no research to support that idea.
Anyhow, one of the Big Ideas people often use to improve the situation is pre-registration of studies. I feel like I see someone suggest this at least once a week. Best example is this one. I personally don't see how this could possibly work.
In my experience, probably about 1 in 3 or 1 in 4 studies that I initiate wind up being published. And I think this is probably close to an optimal number. More often than not, we start collecting data and things look like it's not worth pursuing.
Sometimes the results look a lot more confusing. Sometimes the results look great, but in the meantime, we've thought of an awesome tweak, or even major improvement. Sometimes we think of a totally different experiment that is much cooler. Since an experiment in my field takes minimum 6 months of a person's life away, it sucks to do something that isn't the coolest possible idea you have.
Sometimes we find out a colleague is working on something similar. Sometimes there's a big meeting we want to present a certain story at. Sometimes the data tell us our theory was dumb and suggest a better theory.
In all these cases, taking the time to publish a negative result is not worth the cost. Either to me or to the field. But pre-registering studies would force us to spend that time. It would greatly impede the gradient descent process that characterizes our search for cool results.
And I dont know what it would add. Publishing negative results is great in theory, but I think reading a paper that summarizes a negative result would be almost a waste of time. This is getting long... I will explain why I am skeptical about negative results in my next post.
This may be highly field specific. In clinical studies, sure, go for it, pre-register. In my particular tiny little area, no way. I dont know how other areas work, but I suspect most scientists have a similar gradient descent procedure. In fact, in my field, the studies are quite involved, take ~6 months of hard work to collect the data for, and require big preparations. I think we are near to the clinical study end of the spectrum. I suspect that in fields with quicker studies, pre-registering is even more of a non-starter.
Neuroskeptic's post was way tl;dr for this time in the morning, but aren't studies essentially pre-registered when the grant gets funded? Go to RePORTer, put in your keywords--boom, there are all the people planning experiments on your gene or whatever.
ReplyDeleteFor me the grant application has the same relation to the studies done as a Monet painting has to an actual garden. It's impressionistic.
DeleteHi, thanks for this post. I see your point but I also think your idea of pre-registration is perhaps a little rigid, and I also think you may be conflating pre-registration of research with publication of negative results.
ReplyDeleteIt would quite possible to have a pre-registration mechanism that doesn't mandate publication - but which does allow scientists and the public to keep track of research that was started but not published (or abandoned, or whatever). Would be straightforward to update the status of the pre-registered project accordingly if, say, you thought of a better idea along the way.
I'm a big fan of pre-registration. It's essential in the clinical world and there is no good reason IMO for it not to be applied to other fields. In case you're interested I wrote about this yesterday, see point 3 especially:
http://neurochambers.blogspot.co.uk/2012/07/the-dirty-dozen-wish-list-for.html
I agree about clinical world. But for basic research I don't get it. If you don't compel me to publish it, what incentive do I have not to have my lab manager pre-register every lame idea I come up with (about 10/day personally), just in case it comes up awesome?
Delete