What does “scientists say” mean?
This article introduces my “How Do We Know?” series. In each How Do We Know? post, I will take on a topic where science intersects with contemporary conversations—issues such as climate change, vaccines, the age of the universe, etc.—and address how we know what we know about issues such as these.
I’ve always been as interested in how we know as in what we know. Statements like “scientists say that the polar ice caps are melting,” or “scientists say vaccines are safe and effective”—statements that claim to tell us what we know—make me ask how we know. Maybe you have the same reaction. How do we know that the icecaps are melting? How do we know that vaccines are effective? What is the evidence? How was it analyzed? How were the conclusions reached? Who are the scientists, and who funded their research?
To answer such questions, we need to understand a bit about how science works, not as it is taught in science textbooks but as it is actually practiced. That is the topic of this post.
How science works
First, scientific ideas must be tested against evidence. New scientific insights often come from unconventional thinking, but if allowed every unconventional statement were allowed into the scientific literature, science would be a chaotic mess. Out-of-the-box ideas must be tested. Many creative scientific ideas fail when they crash into the hard reality of evidence. (Spectacular crashes include cold fusion, polywater, and arsenic DNA, although the latter is still being debated.)
Evidence may come from experiments for fields where experiments are possible (much of biology, chemistry, and physics), or from observations in fields such as astronomy, geology, or paleontology where experiments are not possible. (Astronomers cannot create new stars under different conditions to discover how stars form. They must observe stars forming by detecting signals through telescopes. Paleontologists cannot rewind the evolutionary clock to see what would happen with different starting conditions. They must observe the evidence in the fossil record and in the DNA of present species.)
In addition to being tested against evidence, new ideas must also conform to major, well-tested scientific theories: the theories of relativity, quantum mechanics, chemical reactivity, and, yes, evolution. It is true that on rare occasions new ideas show that a prevailing scientific theory is incomplete, but those are usually once-in-a-generation findings. Nearly all new observations fit into the prevailing scientific framework, and if they don’t it’s almost always because someone made a mistake somewhere.
In fact, most scientists are very hesitant to suggest that an established scientific theory should be overthrown by their new idea. Those who make such suggestions too readily are known as crackpots and cranks. Scientific revolutions are rare indeed. (Meetings of major physics organizations sometimes have sessions set aside for science that is out of the mainstream. This avoids the conference organizers having to justify turning down abstracts submitted for presentation and allows an outlet for some of the wilder ideas out there. Put the crackpots into a room together and let them talk to each other.)
Finally, after a scientific idea has been tested against evidence and against major, well-tested theories, it must be presented to a wider scientific audience. This usually happens through publication in scientific journals. A good scientific publication will require that the methods of data collection and analysis be clearly spelled out. It will also specify the funding sources for research so that any potential bias related to the source of funding can be evaluated.
Before an article with new findings is accepted for publication in a respectable scientific journal the manuscript is sent to peers, other scientists working in the same field, for review. These scientists (three or more for good journals) examine the evidence and the analysis presented to see if the evidence is legitimate and the analysis valid. Peer review for well-respected journals can be rigorous. I know! Most manuscripts I have submitted for publication have come back from the journal editor with requests for revision, sometimes substantial. Usually, the request is for further justification of the conclusions drawn, further analysis of data, or even more experiments.
Almost invariably peer review results in a better paper. The method is not foolproof, but it is an important check. The system works remarkably well. Still, occasionally bloopers make their way through the process. (See, for example, this paper on water memory in the journal Nature, which contradicts the known physics of water. It’s a spectacular—and rare—case given that Nature is probably the most prestigious of all scientific journals.) Intentional scientific fraud—the publication of fabricated data—also occurs, although rarely. That subject must be left for another post.
By presenting their work at scientific conferences and subjecting it to peer review, scientists make their findings and interpretations available to the scientific community and the public at large. Philosopher of science Michael Polanyi argued that this is what is meant when we assert that a result is objective. Even though all knowledge, including scientific knowledge, is unavoidably personal (Polanyi’s magnum opus is entitled Personal Knowledge), the scientist committed to a finding presents it to the world with “universal intent.” Universal intent means that, once published, the finding is out there for anyone to read, scrutinize, and test. Results or interpretations that are incorrect, if they are important for the advancement of the field, will be tested and corrected in the research work of other scientists. The scientific literature is replete with examples of previous results corrected or reinterpreted.
Evaluating scientific journals
A further note on scientific publications: You cannot trust an article simply because it is published by a scientific journal. In recent years the number of scientific journals has mushroomed. Anyone can start a scientific journal (or hundreds of them!) by simply setting up a website. Some of these are known as “predatory journals”—journals that will publish almost anything if the authors are willing to pay fees that may reach up to thousands of dollars. (You can find one list of suspected predatory journals here.) The articles published by some of these journals are of varied quality.
The reputation of the journal matters! Journals published by professional societies (American Physical Society, American Chemical Society, Biophysical Society, Federation of European Biochemical Societies, etc.) are typically reputable, as are journals published by major publishing companies such as Springer and Elsevier. It can be difficult for a non-scientist to know whether a journal is reputable or not. One way to start is to check if the journal is indexed by scholarly databases such as PubMed (administered by the National Library of Medicine), SciFinder (administered by the Chemical Abstracts Service of the American Chemical Society), or Web of Science.
A word about the internet: You cannot trust what you find about science on the internet any more than on other topics! Much of what parades as scientific findings on the internet is grossly exaggerated or simply made up. If you cannot trace scientific claims back to a reputable, peer-reviewed journal, then you should be highly suspicious of the claim.
So that’s a summary of how the scientific process works, how the sausage is made. It’s not a perfect process (it is carried out by humans, after all), but it works remarkably well. So when you read or hear a phrase like “Scientists say …,” ask yourself some questions. What is the evidence? Who are these scientists? (There is no uniformly thinking group of “scientists” of whom we can say “scientists think,” whatever the topic may be.) Have they published their conclusions in a reputable journal? Who funded their research?
In future posts I will ask these questions about specific, pertinent scientific issues to describe how we know what we claim to know.
Next up: How do we know that global warming is caused by humans?