SICP: Scientific Information Communication Protocol

A researcher does a study. Based on the results, he comes to a conclusion about a small part of the nature of the universe. He writes up this conclusion and sends it to a journal. His peers review it and conclude that he did a good job in designing, implementing, and interpreting his study. In due time, his paper is published. The researcher receives non-monetary rewards.

Then, one day, he realizes he was wrong. He was wrong in a small way; he doesn't have to do another study to disprove his results, but he realizes he leapt to his conclusions. His interpretation was faulty. His data would have appeared different under very slightly changed study design. He considers letting his colleagues know. But how? He can't amend his paper. It is out there, as unrecallable as toothpaste from a tube. Does he write a letter to the journal? Post a note on his web page? Perhaps blog? None of these things will reach the audience that his original paper did.

It struck me, recently, how clumsy the communication protocol is that researchers use. They publish in papers. For a paper to change after publication, a slow, complicated machinery must be put in motion. A subsequent paper can show conflicting results. The paper can be found to be based on fraudulent data and recalled. Or the results can simply be gradually forgotten as science moves in a different direction. In any of these situations, change is slow.

I said as much in my bioethics class today. The professor asked, "How would you change things?" I suggested a world in which peer reviewed papers still existed, but another, non-peer-reviewed channel existed as well. We'd all know to take this channel's contents with a grain of salt, as they would be offered without the cautious filter of peer review. We would be much less likely to reference data from this channel in our papers. But we'd keep abreast of the conversations in it before completely trusting information we found in the more respectable peer-reviewed channel. It would be a place for back-and-forth, for something closer to a true conversation. (I've seen conversations in journals: two labs publishing an alternating series of articles that reference and refute each other, a dialog conducted over the course of years. It's rare and somewhat awestriking to observe.)

One of my classmates replied that I was idealistic. Well, sure. But this second channel is technologically very possible today. It might even currently be in the midst of developing out of the medium of science blogging. What would it take to start building mechanisms for allowing people to search it as easily as they currently use PubMed and Google Scholar to search the primary channel? I can't yet think where to start. Maybe all I need to do is sit back and wait, and watch the scientific information communication protocol evolve on its own, from its current clumsy elephantine form to something more nuanced and potentially graceful.