This seems obvious, but in practice, a lot more work goes into original studies designed to create interesting conclusions than into the rather less interesting work of reproducing studies that have already been done to see whether their results hold up.
That's why efforts like the Reproducibility Project, which attempted to retest findings from 100 studies in three top-tier psychology journals, are so important. As it turns out, findings from the majority of the studies the project attempted to redo could not be reproduced. The New York Times reported:
Now, a painstaking yearslong effort to reproduce 100 studies published in three leading psychology journals has found that more than half of the findings did not hold up when retested. The analysis was done by research psychologists, many of whom volunteered their time to double-check what they considered important work. Their conclusions, reported in the journal Science have confirmed the worst fears of scientists who have long worried that the field needed a strong correction.This is a serious problem for psychology, and for social science more broadly. And it's one that, as the Times points out, observers in and around the field have been increasingly worried about for some time.
Why is psychology research (and, it seems likely, social science research generally) so stuffed with dubious results? There are three likely reasons:
A bias towards research that is not only new but interesting: An interesting, counter-intuitive finding that appears to come from good, solid scientific investigation gets a researcher more media coverage, more attention, more fame both inside and outside of the field. A boring and obvious result, or no result, on the other hand, even if investigated honestly and rigorously, usually does little for a researcher's reputation. The career path for academic researchers, especially in social science, is paved with interesting but hard to replicate findings. (In a clever way, the Reproducibility Project gets around this issue by coming up with the really interesting result that lots of psychology studies have problems.)
An institutional bias against checking the work of others: This is the flipside of the first factor: Senior social science researchers often actively warn their younger colleagues—who are in many cases the best positioned to check older work—against investigating the work of established members of the field. As one psychology professor from the University of Southern California grouses to the Times, "There’s no doubt replication is important, but it’s often just an attack, a vigilante exercise."
This is almost exactly what happened in an incident earlier this year when a couple of grad students first started to find discrepencies in a major study about attitudes toward gay marriage. The study, which claimed to find that attitudes on gay marriage could be quickly made more positive by a 20 minute chat with someone who is gay, turned out to be built on fake data. The grad student who uncovered the fakes has said that, over the course of his investigation, he was frequently warned off from his work by advisers, who told him that it wasn't in his career interest to dig too deeply.
Small, unrepresentative sample sizes: In general, social science experiments tend to work with fairly small sample sizes—often just a few dozen people who are meant to stand in for everyone else. Researchers often have a hard time putting together truly representative samples, so they work with subjects they can access, which in a lot of cases means college students.
One of the most controversial topics today is global warming. To critics, it is a fraud and shame. To proponents, it is the most important discovery since the big bang. The problem with the billions of dollars spent on global warming studies is that the results are based upon "models" created by the researchers which discount actual, verifiable facts like actual temperature readings. The models are created to make assumptions, which can be manipulated by the type of data used in the pre-determined formulas to get a pre-defined result.
In elementary school, we learned about the planet. It's environment had changed over time. There was a massive ice age which covered much of the planet. Then a massive warming spell that carved the Great Lakes from the receding ice flows. Montana used to be a tropical jungle when the dinosaurs roamed the planet. And all of these great climate changes had nothing to do with mankind's industrialization of the planet. The Earth's climate is really determined by the radiation from the sun, the ozone layer which reflects much of the radiation, the tilt of the planet's axis, rotation of the planet around the sun and moon, and ocean currents which are the engine for all weather patterns.
The problem with global warming research is that various scientific groups cannot verify results because the inherit flaws in the proprietary models used by the various groups. The result is like two children arguing whether the sky is white or gray.
True science has been a major influence on the quality of life for us. Real, verifiable and repeatable research has led to life changing medicines, vaccines and inventions which have enriched our lives. But there is an alarming trend of pseudo-science and academic fraud which diverts valuable resources from worthy research efforts. Our we getting lazy? By getting grant money for projects in which we rig the results beforehand in order to appease a political viewpoint is the new norm, are we dooming ourselves to a state of lethargic, intellectual appeasement?
