The 8 Big Ideas:

• Hawthorne Effect: People Behave Differently When Being Observed

• Knowledge Is Built On Prior Knowledge

• We Take Too Many Things On Faith

• Knowledge Exists but Remains Unconnected

• Why Most Research Findings are Overstated

• Replication Uncovers False Knowledge

• Change Blindness

• As We Get Older We Lose The Drive To Learn Broadly And Become Ever More Focused On One Thing

My Highlights From the Book:

Hawthorne Effect: People Behave Differently When Being Observed:

This is similar to the well-known Hawthorne effect, when subjects behave differently if they know they are being studied. The effect was named after what happened in a factory called Hawthorne Works outside Chicago in 1920s and 1930s. Scientists wish to measure the effects of environmental changes, such as lighting, on the productivity of the workers. They discovered that whatever they did to change the workers behaviors – whether they increase the lighting or altered in your aspect of their environment – resulted in increased productivity. However, as soon as the study was completed, the productivity dropped.

The researchers concluded that the observations themselves were affecting productivity and not the experimental changes. The Hawthorne effect was defined as “an increase in worker productivity produced by the psychological stimulus of being singled out and made to feel important.” While it has been expanded to meet any change in response to being observed and studied, the focus here on productivity is important for us: if the members of an industry know that they are being observed and measured, especially in relationship to a predicted metric, perhaps they have an added incentive to increase productivity and meet the metrics expectations.

Knowledge Is Built On Prior Knowledge:

Why else would everything be adhering to these exponential curves and growing so rapidly? I like the answer is related to the idea of community knowledge. Anything new – an idea, discovery, or technological breakthrough – must be built upon what is known already. This is generally how the world works. Scientific ideas build upon one another to allow for new scientific knowledge and technologies and are the basis for new breakthroughs. When it comes to technological and scientific growth, we can bootstrap what we have learned before toward the creation of new facts. You must gain a certain amount of knowledge in order to learn something new.

We Take Too Many Things On Faith:

We can look to the children’s game of telephone to understand how facts can be corrupted and spread: the children sit in a circle, and one person begins by whispering a phrase or sentence to the child next to them… But in general, the sentence decays without any malice or intent. It simply gets changed because hearing a whispered sentence doesn’t provide great fidelity. It’s what information scientist referred to as a noisy channel. But information is passed from one person to another it has the potential to become inaccurate unless there are a whole host of error checking mechanisms.

Luckily, there is a simple remedy: be critical before spreading information and examine it to see what is true. Too often not knowing where once facts come from and whether it is well-founded at all is the source of an error. We often just take things on faith.

Knowledge Exists but Remains Unconnected:

What does it mean for knowledge to exist but remain hidden? It’s one thing for a result to be ignored – that’s a specific instance of this sort of knowledge, which will be discussed later. But Swanson was talking about research that, due to his inability to combine with other findings, is less valuable than it could be. Imagine that in one area of the scientific literature there was a paper showing that A implies B. Then, somewhere else, some seldom read Journal, an article contained the finding that B implies C. but since no one has read the papers, the obvious result – that A implies C – remained dormant, hidden in the literature as an unknown fact.

Hidden knowledge takes many forms. At its most basic level hidden knowledge can consist of pieces of information that are unknown, or are known only to a few, and, for all practical purposes, still need to be revealed. Over time hidden knowledge includes facts that are part of undiscovered public knowledge, when bits of knowledge need to be connected to other pieces of information in order to yield new facts. Knowledge can be hidden in all sorts of ways, a new facts can only be created if this knowledge is recognized and exploited.

Why Most Research Findings are Overstated:

John Ioannidis is a Greek physician and professor at the University of Ioannina school of medicine, and he is obsessed with understanding the failings and more human properties of the scientific process. Rather than looking at anecdotal example, that is the case of though, aggregates many cases together in order to paint a clear picture of how we learn new things in science. He has studied the decline affect himself, fighting its consistent presence within the medical literature. It is found that for highly cited clinical trials, initially significant margin facts are later found to have smaller effects or often no effect at all in a nontrivial number of instances.

Essentially, any quantitative way, the shows that in large numbers of situations – whether due to the study being done in a field in which the above ratio is fairly low, implying that the probability of a spurious relationship is high, or an experiment using very few subjects, or the study was done in an area where replication of results doesn’t occur – physically significant and publishable results can occur, even though they are actually not true.

The smaller the studies conducted in a scientific field, the less likely the research findings are to be true. If the study is small, it can yield a positive result more easily than random chance. This is like the classical clinical trial joke, in which, upon testing a new pharmaceutical on a mouse population, it was reported that one third responded positively to the treatment, one third had no response, and the third mouse ran away.

The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. If an effect is small, it could be like Planet X, and we’re simply measuring noise.

The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true. More experiments mean that some of them might simply be right due to chance, and get published.

The greater the flexibility and designs, definitions, outcomes, and analytical modes anything to the field, the less likely the research findings are to be true. There is a greater possibility of massaging the data to get a good result, then there is a greater chance that someone will do so.

The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true. The scientists are people too, and are not perfect being, greater the possible bias, the greater the chance of finding aren’t true.

The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true. More teams mean that a positive result is a great deal of hype quite rapidly, and is pushed out the door quickly, but leads to research that can be easily refuted, with an equal amount of hype. Ioannidis refers to this as a cause of the Proteus phenomenon, which he defined as “rapidly alternating extreme research claims and extremely opposite refutations.”

Replication Uncovers False Knowledge:

One simple way to minimize a lot of this trouble is through replication, measuring the same problem over and over. Too often it’s much more glamorous to try to discover something new is simply to someone else’s experiment a second time. In addition, many scientists, even those who want to get findings, find it difficult to do so. That’s when they think of result is actually wrong, there is even more of a disincentive.

Why is this so? Regarding a kerfuffle about the possibility of bacteria that can incorporate arsenic into their DNA backbone – a paper published in Science – Carl Zimmer explains:

But none of those critics had actually try to replicate the initial results. That would take months of research: getting the bacteria from the original team of scientists, rearing them, setting up the experiment, gathering results in interpreting them. They scientists are leery of spending so much time on what they consider a foregone conclusion, graduate students are reluctant, but they want their first experiments to make a big splash, not confirm what everyone already suspects.

“I’ve got my own science to do,” John Helmann, a microbiologist at Cornell and a critic of the Science paper, told Nature.

But only through replication can science be the truly error-correcting enterprise that it is supposed to be. The application allows for the overturning of results, as well as an approach toward truth, and it is what science is ultimately about. That paper that followed up on Ioannidis’s somewhat pessimistic conclusion, researchers concluded that a small amount of replication can lead us to much more robust science. But how do we do this?

What can be measured, and when, affects what can be learned. If we can’t measure something, this can actually create a bias in what we know. For example, in biology, there’s something known as taxonomic bias. This is when Eddie’s certain living things not because they are more prevalent but because we like them more, or because they are simply easier to find. Vertebrates – of animals that have backbone enterprise most of the creatures we are familiar with – so the subject of the vast majority of scientific papers, despite being only a tiny fraction of the different types of animals on earth. Sometimes, when this seems overly malicious – amphibians and reptiles getting less attention than birds and mammals, because they are slimy or an alleged left cubby – some scientists even call it taxonomic chauvinism.

What we study is not always what is actually out there; it’s often what we are interested in, or what’s easiest to discover.

Change Blindness:

Change blindness in the world of facts and knowledge is also a problem. Sometimes we are exposed to new facts and simply filter them out. But more often we have to go out of our way in order to learn something new. Our blindness is not a failure to see the new fact; it’s a failure to see that the facts in our minds have the potential to be out-of-date at all. It’s a lot easier to keep on quoting a fact you learned a few years ago, after having read it in a magazine, than to decide it’s time to take a closer look at the current ten largest cities in the U.S., for example, and notice that they are far different from what we learned when we were younger.

As We Get Older We Lose The Drive To Learn Broadly And Become Ever More Focused On One Thing:

The results of this experiment are not terribly surprising. In addition to all of the cognitive biases that we are saddled with, it is difficult for us to keep abreast of all the information around us. When we are young, we are treated as little generalists, absorbing all manner of information. We learn geography, history, mathematics, how to read a map, and lots of science trivia. We are even able to learn entire languages relatively effortlessly.

But then, as we get older, a curious thing happens with our approach to education. In addition to no longer being compelled to learn all manner of things (because we are, after all, adults, and we really can’t be compelled to learn anything at all), if we do continue to educate ourselves, we focus. We choose a major and learn all that there is to learn about a single topic, such as biology. Then we become experts in that area, well aware of all the nuances, debates, and changes in facts within that field. We learn more and more about less and less.

But all of our earlier knowledge remains in statis. Instead of it all growing and developing in a rigorous fashion, like whatever we choose to make our careers, it generally stays the same. Unless we happen to stumble upon an article in a magazine or newspaper about a certain scientific finding, or unless something is so important and earth shattering can’t help but remark upon this new fact’s novelty, we remain stuck at the factual level of our grade-school selves.