Would pre-registering studies really work?

The zeitgeist is that science is deeply corrupt and needs to take major steps to improve its quality.

I agree that there are problems in science, although I think (following research by Dan Ariely) that major cheaters are relatively rare and that minor cheating is near universal. I also doubt that cheating has increased, but I have no research to support that idea.

Anyhow, one of the Big Ideas people often use to improve the situation is pre-registration of studies. I feel like I see someone suggest this at least once a week. Best example is this one. I personally don't see how this could possibly work.

In my experience, probably about 1 in 3 or 1 in 4 studies that I initiate wind up being published. And I think this is probably close to an optimal number. More often than not, we start collecting data and things look like it's not worth pursuing.

Sometimes the results look a lot more confusing. Sometimes the results look great, but in the meantime, we've thought of an awesome tweak, or even major improvement. Sometimes we think of a totally different experiment that is much cooler. Since an experiment in my field takes minimum 6 months of a person's life away, it sucks to do something that isn't the coolest possible idea you have.

Sometimes we find out a colleague is working on something similar. Sometimes there's a big meeting we want to present a certain story at. Sometimes the data tell us our theory was dumb and suggest a better theory.

In all these cases, taking the time to publish a negative result is not worth the cost. Either to me or to the field. But pre-registering studies would force us to spend that time. It would greatly impede the gradient descent process that characterizes our search for cool results.

And I dont know what it would add. Publishing negative results is great in theory, but I think reading a paper that summarizes a negative result would be almost a waste of time. This is getting long... I will explain why I am skeptical about negative results in my next post.

This may be highly field specific. In clinical studies, sure, go for it, pre-register. In my particular tiny little area, no way. I dont know how other areas work, but I suspect most scientists have a similar gradient descent procedure. In fact, in my field, the studies are quite involved, take ~6 months of hard work to collect the data for, and require big preparations. I think we are near to the clinical study end of the spectrum. I suspect that in fields with quicker studies, pre-registering is even more of a non-starter.

The hottest day of the year

Remember seeing this?

And remember when walking to work was like this?

Remember when you DIDNT want to just jump right into the lake??

Even going out of doors for a moment was a trudge??

It seems so very far away. 

But it cools me off just to look at these pictures.

Does Frontiers want to be taken more seriously?

I love Frontiers in theory, I honestly do.
There are a lot of great things about it.

But I think they can't be taken 100% seriously unless they take some steps to improve their image.

Basic stuff to make them seem like a legitimate journal, not something someone did over spring break.

I would recommend the following 7 things:

1. Make the website a lot faster.

2. Hire someone to organize it logically. It's nearly impossibly to find things one wants. I'm thinking they could use the Google+ system, since no one else is using it. That way they'd have a search function that works. But also, they need to hire someone who has a sense of website navigation. It's almost comical how hard it is to find the thing you want.

3. Avoid mission creep. As Frontiers keeps expanding, it seems more and more like a Ponzi Scheme, expanding into new areas as old ones wither. This would also reduce clutter on the website and make it easier to find things.


4. And on the mission creep domain, Stop trying to be Facebook. Scientists already have a social network. It's called Facebook.

5. Stop spamming reviewers all the time. Especially with articles that are distant from anything we know about.

6. Give up on the 'interactive review' theory. Most people I have talked to informally write a normal review, put all the comments in one box, and ignore the rest. And then get really annoyed with more than one extra round. We are giving out time to review. We don't want to have a conversation.

7. I dont know if it's just me but there seem to be cookie problems. Frontiers never remembers that I am logged in, and asks for my password every time I go there. No idea why. I dont have this problem with most other websites I use.

Thoughts on turning 1

I am the only professor in the department who doesn't know his/her office phone number.

My phone is in the drawer. Call my cell phone.

I refuse to use the University email system. I still use gmail exclusively.

I am the newbie, so I am the only professor in the dept who has to park a zillion miles away and walk.

I am the only professor who doesn't have a desktop computer. Even though they'd let me put in on my startup, I don't want one. Too much hassle.

As far as I can tell, I have the best summer attendance record of the faculty in the department. Not sure that's a good thing.

I'm the only person who has never been on a faculty search committee, so I am the only one who is excited to see how that particular sausage gets made. Everyone else is jaded, probably justifiably, and laughs at my enthusiasm.

I'm the only person I know who brings his dog to his office. I think I might be getting dirty looks when I do that.

I need a good book on the Theory of Mentoring.

I get a tiny thrill when undergraduates call me "Professor." Although a pedantic voice in my head wants to correct them and say "Technically I am an Assistant Professor."

I still don't have the whiteboard in my office I asked for a year ago. I wonder where it is. Probably time to nag someone.

I still have not been propositioned by an undergraduate in exchange for a grade, slept with a student, fought viciously with anyone over space, written a devastating book review of the guy next door, or spent a whole semester coming in to work and sleeping in my office all day, or any of the other stereotypical things professors do in movies. I don't have elbow pads or a tweed anything.

I am the only person in the department with a protocol to do research in Puerto Rico. This will be useful in February when it is freezing cold here and 85 degrees in PR.

First Birthday!

I have been a baby professor for one year!

Solution

Why do MBA types use the word "solution" all the time?

First I thought it sounded ridiculous, but then I realized, might as well lean into it.

So on m CV, instead of "manuscripts in prep" now it says "solutions in prep".

And my spring course, instead of being called "laboratory in neurophysiology" will be called "Workshop in Neurosolutions"

And my lab meeting is called "Solutions Breakout Session".

Any my R01, instead of "Background, Significance, Methods" is divided into "Previous solutions, Strategic Dynamic Solutions, and Solutions."

Some famous TV Shows are structurally superhero narratives

Dexter
Hero: Dexter
Superpower: Super-fighting
Secret Identity: Dexter, the nerdy detective
Weakness: Psychopath
Sidekick: Harry's Ghost

Breaking Bad
Hero: Walter
Super-power: Super-chemistry
Secret Identity: High School teacher
Weakness: Anger issues
Sidekick: Jesse

Mad Men:
Hero: Don Draper
Super-power: Super copy-writing
Secret Identity: The man in the gray flannel suit
Weakness: Existential ennui
Sidekick: Roger

Sopranos
Hero: Tony
Super-power: Good at running a criminal organization
Secret Identity: Law abiding citizen
Weakness: Needs psychiatric help
Sidekick: Christopher

Veronica Mars
Hero: Veronica
Super-power: Super-sleuthing
Secret Identity: Mild mannered high school student
Weakness: Logan
Sidekick: Her father

Big Love
Hero:  Bill
Super-power: Super-polygamy
Secret Identity: Monogamous regular guy
Weakness: Nutty, also, connection to compound
Sidekick: Bud