Stuart Buck with a fascinating piece on the seemingly simple but deceivingly hard process of replicating studies. Two labs were collaborating on a breast cancer study, and ran into surprising challenges getting the same results with (what they thought were) the same inputs:

They were frustrated: “Despite using seemingly identical methods, reagents, and specimens, our two laboratories quite reproducibly were unable to replicate each other’s fluorescence-activated cell sorting (FACS) profiles of primary breast cells.”

They tried everything: The instrumentation. The “specific sources of tissues…, media composition, source of serum and additives, tissue processing, and methods of staining cell populations.” The protocols to ensure they were using “identical enzymes, antibodies, and reagents.”

After doing all of this for a year, they were still stumped. So they met in person to “work side by side so we could observe every step of each other’s methods.”

In the end, they figured out that the ONLY reason for their discrepant results was this: The rate of stirring collagenase. At one lab, the tissue was stirred at a rate of 300 to 500 revolutions per minute for six to eight hours. At the other lab, it was stirred at a much more gentle rate of 80 revolutions per minute for 18 to 24 hours.

That was it. That was the one and only difference that explained why Harvard and Berkeley labs were getting such different results from an identical experiment. No one had even thought to mention the rate of stirring, because it seemed so routine and unlikely to matter.

There’s already a replication crisis in research. This shows that even when one is trying really hard to replicate honest research, we can be confounded by intricate minutiae of process and incidental detail. It all matters.

You’ve also gotta read it for the hot dog story.