The Core Theme - Knowledge and the Knower
Lesson 5 - Method and Tools 1: How We Claim to Know What We Know.
Being wrong for a long time is not the same as being stupid. Understanding why intelligent people get things wrong is as important as understanding how to get things right.
Lesson 5 - Method and Tools 1: How We Claim to Know What We Know.
Being wrong for a long time is not the same as being stupid. Understanding why intelligent people get things wrong is as important as understanding how to get things right.
THE PROVOCATION
Were They Wrong, or Were They Lying?
The experts weren't lying. They were wrong in a way that turns out to be extremely interesting.
Were They Wrong, or Were They Lying?
The experts weren't lying. They were wrong in a way that turns out to be extremely interesting.
|
|
Dr. Anthony Fauci, March 2020
60 Minutes In March 2020, Dr. Anthony Fauci, Director of the NIAID, became the scientific face of the U.S. government’s COVID-19 response. As a lead member of the White House Coronavirus Task Force, he utilized daily televised briefings to explain complex viral dynamics to a panicked public. In this clip from March 2020 he advises about the use of face masks in public spaces. |
You just watched it. In March 2020, the most senior infectious disease official in the United States told the world there was no reason to wear a mask. The WHO said the same. So did the UK government, the French government, the Swiss government and virtually every other scientific establishment in the world. This was not a fringe position. It was the consensus of the best-credentialed experts in the world. Six weeks later, they reversed that decision, now masks were essential. The guidance that had been given to billions of people was wrong.
The question this lesson begins with is not whether they were wrong. Of course they were wrong. They were doing what all knowers do, making the best inference available from incomplete evidence. They were being rational, but they were still wrong.
Understanding how this is possible, how intelligent, highly-qualified, well-intentioned experts can be systematically wrong, and why correcting them takes so long, is the central question of western epistemology. It is also one of the most practically important questions you will encounter in your lifetime. The tools for thinking about it were not developed in 2020. They were developed over three centuries, and one of the clearest illustrations comes not from a pandemic laboratory but from history and a street map drawn by a London anaesthesiologist in 1854. That map is where this lesson begins, because before we can ask how knowledge is justified, we need to see what justified knowledge actually looks like when it works, and what tools made it possible.
Understanding how this is possible, how intelligent, highly-qualified, well-intentioned experts can be systematically wrong, and why correcting them takes so long, is the central question of western epistemology. It is also one of the most practically important questions you will encounter in your lifetime. The tools for thinking about it were not developed in 2020. They were developed over three centuries, and one of the clearest illustrations comes not from a pandemic laboratory but from history and a street map drawn by a London anaesthesiologist in 1854. That map is where this lesson begins, because before we can ask how knowledge is justified, we need to see what justified knowledge actually looks like when it works, and what tools made it possible.
Before we begin: The language of arguments
A Toolkit for Evaluating Claims
Before examining how scientific reasoning works, it helps to have precise vocabulary for what counts as a good argument. These three terms function as analytical tools - instruments for examining the structure of claims rather than just their content. They will appear throughout the rest of the course, particularly in Social Sciences, where the quality of reasoning behind a claim is always the first question to ask.
A Toolkit for Evaluating Claims
Before examining how scientific reasoning works, it helps to have precise vocabulary for what counts as a good argument. These three terms function as analytical tools - instruments for examining the structure of claims rather than just their content. They will appear throughout the rest of the course, particularly in Social Sciences, where the quality of reasoning behind a claim is always the first question to ask.
|
VALID ARGUMENT A valid argument is one where the conclusion follows necessarily from the premises. If the premises are true, the conclusion cannot be false. Validity is about logical structure, not about whether the premises are actually true. Example: All students at Ecole Moser speak at least two languages. You go to Ecole Moser. Therefore you speak at least two languages. - This is valid. If both premises are true, the conclusion must be true. Counter-example (valid but unsound): All students at Ecole Moser have two heads. You go to Ecole Moser. Therefore you have two heads. - This is also valid. The logical structure is identical. But one premise is false, so the conclusion is false. Validity does not guarantee truth. |
INVALID ARGUMENT An invalid argument is one where the conclusion does not follow from the premises, even if the premises are true. The classic mistake: running the logic backwards. Example: All students at Ecole Moser speak two languages. You speak two languages. Therefore you go to Ecole Moser. This is Invalid. Millions of people speak two languages. The conclusion does not follow. This error - treating a sufficient condition as if it were a necessary one - is one of the most common formal mistakes in everyday reasoning. Sound A sound argument is both valid AND has premises that are actually true. Soundness is the gold standard: if an argument is sound, its conclusion is guaranteed to be true. |
When scientists told the public in early 2020 that masks were not necessary, the argument was inductive, not deductive. The premises were: available evidence suggests droplet transmission dominates; therefore masks for the general public offer limited protection. The conclusion did not follow with logical necessity - and it turned out to be wrong. Understanding the difference between a valid argument and a strong inductive one is central to evaluating any knowledge claim. Use these terms as tools. When you encounter a claim, ask whether the reasoning is valid, whether the premises are true, and whether the argument is sound.
Four big ideas about how we claim to know what we know.
Big idea 1 - Evidence does not speak for itself
The same facts are compatible with more than one explanation. Knowing which explanation is right requires something beyond the facts alone, and that something is harder to find than it looks.
Big idea 1 - Evidence does not speak for itself
The same facts are compatible with more than one explanation. Knowing which explanation is right requires something beyond the facts alone, and that something is harder to find than it looks.
This lesson should really be about the scientific method, but because I'm a history teacher it isn't; at least not directly. Instead, it combines two of my favourite things Victorian Britain and the writer and Steven Johnson (Mr. NotebookLM). In fact, I do address the background of this in a lesson in DP1 but you probably missed it. But this time its unavoidable because I'm going to integrate this story into each of the four big ideas that make up this section. But first the story.
"He reviewed Farr's list again, looking this time for telltale absences."
Steven Johnson, The Ghost Map (2006)
Steven Johnson, The Ghost Map (2006)
|
|
Steven Johnson is a prominent media theorist and author whose work explores the history of innovation. In his acclaimed book The Ghost Map, he chronicles the 1854 London cholera outbreak, and today, Johnson applies these principles of "networked thought" as the Editorial Director of NotebookLM at Google. He helped develop the AI tool to serve as a collaborative research assistant, empowering users to synthesize complex information and generate new insights, and allowed me to make many podcasts and infographics on this website.
|
|
In the late summer of 1854, a disease killed 616 people in ten days in a single neighbourhood of London. Nobody knew what caused it. The city's leading medical authorities were certain of the answer: miasma, bad air rising from the sewers and the Thames. This was not a fringe theory. Edwin Chadwick, who had done more than any person alive to improve London's public health infrastructure, was a passionate miasmatist. Florence Nightingale was a miasmatist. The General Board of Health was staffed by miasmatists. The consensus was overwhelming.
It was also completely wrong. The cause was contaminated water from a single pump on Broad Street. A methodical anaesthesiologist named John Snow proved it using a map, this little map on the right. You should click on it and enlarge it. It is one of the most important maps ever made. |
|
Snow plotted every death on a street map of the neighbourhood. The pattern was unmistakeable. The deaths clustered around the pump. But the map alone did not prove his theory - the miasmatists could argue that the same area had the worst-smelling air in the district, and that the bad air, not the water, explained the cluster. The same data can, in principle, support more than one account of what caused it.
The Tools of Snow's Investigation
The Bills of Mortality |
Pre-existing parish death records that gave Snow his raw data. These were administrative tools repurposed for epidemiological inquiry - Snow did not collect new data from scratch, he analysed records already being kept for other purposes. |
The street map |
Snow hand-drew a map of the Soho neighbourhood and plotted each death as a dot. This spatial visualisation tool transformed the same data the miasmatists had into a pattern that was visible at a glance. The miasmatists had mortality tables; Snow had a map. The tool changed what the evidence could show. |
The anomalous case as a probe |
Snow did not just look at where deaths occurred. He went looking for places where deaths should have occurred but did not. The workhouse with 535 residents and only two deaths, the brewery where no workers died - these absences were found by using the hypothesis as a search instrument, a directed probe into the data. |
The pump handle |
Snow's eventual intervention was itself a tool for testing. Removing the handle was not just a public health measure - it was an experiment. If deaths fell after removal, the water-pump hypothesis was supported. If they continued at the same rate, it was not. The handle was a physical test of a causal claim. |
Local knowledge |
The Reverend Henry Whitehead, a local clegyman who initially doubted Snow's theory, became one of his most important epistemic tools. Whitehead's intimate knowledge of the neighbourhood helped explain the anomalies - including tracing the index case to a baby whose nappy water had been emptied near the pump. |
What made Snow's investigation decisive was what he did next. Rather than look for further evidence that confirmed he thesis, he went looking for cases that should have appeared on the map but did not. He tried to disprove his theory. Across the street from the pump stood a workhouse with 535 residents. If the water or bad smells was killing people, the workhouse should have accounted for dozens of deaths. There were two. Snow discovered that the workhouse had its own well. Nearby, a brewery employed hundreds of men who worked long hours in the heat. None had died. The brewery provided workers with a daily ration of malt liquor. They never drank from the pump. These absences were as important as the deaths on the map. Snow approached the outbreak with a hypothesis: the cause was contaminated water, and the Broad Street pump was the source. This is what hypothesis-driven inquiry means. Evidence is not simply found. It is produced through a process of directed search, in which the hypothesis determines what counts as a relevant observation. What comes back either supports it or forces it to change.
In early 2020, the hypothesis guiding mask guidance was that respiratory viruses spread primarily through large droplets that fall quickly to the ground and not through fine aerosols that hang in the air. The evidence available was consistent with this picture. What nobody was systematically looking for were the anomalous cases, the superspreader events in poorly ventilated indoor spaces where no one had been close enough for droplet transmission to explain the spread. More on this later.
Big idea 2 - Induction: the engine of knowledge, and its problem
Every claim we make about the world that goes beyond what we have directly observed is an inductive inference. And induction can never be fully justified.
Every claim we make about the world that goes beyond what we have directly observed is an inductive inference. And induction can never be fully justified.
"Hume was perfectly right in pointing out that induction cannot be logically justified. He held that there can be no valid logical arguments allowing us to establish "that those instances, of which we have had no experience, resemble those, of which we have had experience.""
Karl Popper, paraphrasing Hume in Conjectures and Refutations (1963)
Karl Popper, paraphrasing Hume in Conjectures and Refutations (1963)
Again we have encountered the content of this lesson in a history lesson, this time in the far distant past of 11e. But before we go any further, here is a spendid NotebookLM generated infographic to remind you of the difference between inductive and deductive reasoning.
The two great forms of reasoning are deduction and induction. Deduction moves from premises to a conclusion that necessarily follows: if all humans are mortal and Socrates is human, then Socrates is mortal. The conclusion cannot be false if the premises are true. But deduction generates no new knowledge - you can only extract what was already implicit in the premises.
Induction moves in the opposite direction, from particular observations to general conclusions. Snow examined the geographic distribution of cases across two outbreaks and concluded that contaminated water was the cause. Every contact tracer mapping a transmission chain in 2020 was reasoning the same way: these cases share this exposure, so that exposure is probably the source. Every R number and every vaccine efficacy figure was an inductive inference from observed data to a general claim about how the virus behaves. Induction is the engine of empirical knowledge. David Hume, writing in 1748, showed it is also incapable of logical justification.
Induction moves in the opposite direction, from particular observations to general conclusions. Snow examined the geographic distribution of cases across two outbreaks and concluded that contaminated water was the cause. Every contact tracer mapping a transmission chain in 2020 was reasoning the same way: these cases share this exposure, so that exposure is probably the source. Every R number and every vaccine efficacy figure was an inductive inference from observed data to a general claim about how the virus behaves. Induction is the engine of empirical knowledge. David Hume, writing in 1748, showed it is also incapable of logical justification.
|
Hume's argument is simple and devastating. What justifies the inference from past experience to future expectation?
Well it can't be logic because the conclusion does not follow necessarily from the premises. If you try to justify induction by saying that it has worked reliably in the past, you are using induction to justify induction, which is circular. Hume's conclusion: the belief that the future will resemble the past - the foundation of all empirical knowledge - cannot be rationally secured. This is explained by another fantastic NotebookLM infographic right. Click to enlarge! |
|
Hume's problem sits, unresolved, underneath every prediction made during the pandemic. The models that projected mortality, the trials that established vaccine efficacy: all of it rested on inductive inference, on a foundation Hume showed we cannot fully secure. We act as though induction justifies our beliefs because we cannot do without it.
Big idea 3 - Popper's response: the logic of falsification
You cannot prove a scientific theory. But you can disprove one. Thats what gives science its distinctive power.
"It is easy to obtain confirmations, or verifications, for nearly every theory - if we look for confirmations. Confirmations should count only if they are the result of risky predictions; that is to say, if, unenlightened by the theory in question, we should have expected an event which was incompatible with the theory. Every "good" scientific theory is a prohibition: it forbids certain things to happen. The more a theory forbids, the better it is."
Karl Popper, Conjectures and Refutations (1963)
You cannot prove a scientific theory. But you can disprove one. Thats what gives science its distinctive power.
"It is easy to obtain confirmations, or verifications, for nearly every theory - if we look for confirmations. Confirmations should count only if they are the result of risky predictions; that is to say, if, unenlightened by the theory in question, we should have expected an event which was incompatible with the theory. Every "good" scientific theory is a prohibition: it forbids certain things to happen. The more a theory forbids, the better it is."
Karl Popper, Conjectures and Refutations (1963)
Karl Popper's response to Hume was not to deny the problem but to reframe what science is doing. Science should not simply set out to confirm theories, sceince attempts to refute them. The relevant question to ask of any empirical claim is not "what evidence would confirm this?" but "what evidence would falsify it?" A theory that cannot in principle be falsified is not a scientific theory at all. No number of confirming instances can prove a universal claim. Seeing a million white swans supports the claim that "all swans are white". This is an inductive generalisation. For centuries in Europe, all known swans were white. Then in 1697, Dutch explorers led by Willem de Vlamingh arrived in Western Australia and encountered black swans. A single observation overturned a long-standing “truth.” A single black swan refutes the claim with logical certainty. This is why Popper argued that the proper structure of scientific reasoning is not accumulating confirmations but exposing theories to the harshest possible tests.
Falsifiability is a diagnostic tool as well as a logical principle. It allows you to ask of any knowledge claim: what would have to be true for this to be wrong? If nothing could count as evidence against the claim, it is not science. Freudian psychoanalysis, Popper argued, was unfalsifiable in this sense because any behaviour could be reinterpreted to fit the theory, and the theory specified in advance no evidence that would refute it. That did not make it false or useless, it simply made it a different kind of claim from a scientific one. The tool of falsifiability draws a boundary between inquiry that can be corrected by evidence and inquiry that cannot.
Falsifiability is a diagnostic tool as well as a logical principle. It allows you to ask of any knowledge claim: what would have to be true for this to be wrong? If nothing could count as evidence against the claim, it is not science. Freudian psychoanalysis, Popper argued, was unfalsifiable in this sense because any behaviour could be reinterpreted to fit the theory, and the theory specified in advance no evidence that would refute it. That did not make it false or useless, it simply made it a different kind of claim from a scientific one. The tool of falsifiability draws a boundary between inquiry that can be corrected by evidence and inquiry that cannot.
|
|
Popper was born in Vienna and spent most of his academic career at the London School of Economics. His philosophy of science, developed in The Logic of Scientific Discovery (1934) and Conjectures and Refutations (1963), proposed falsifiability as the criterion distinguishing scientific claims from non-scientific ones. His arguments against Marxism and Freudianism as unfalsifiable - theories that could accommodate any evidence by reinterpretation. ( The Open Society and Its Enemies and The Poverty of Historicism)
|
In a way Snow was a Popperian before Popper. He went looking for the cases that should have destroyed his theory. The workhouse with 535 residents and only two deaths was exactly such a case. Snow investigated the anomaly rather than ignoring it, and the explanation - a private well that provided a clean source of water - confirmed the theory precisely by explaining away the apparent counterexample.
In a way Snow was a Popperian before Popper. He went looking for the cases that should have destroyed his theory. The workhouse with 535 residents and only two deaths was exactly such a case. Snow investigated the anomaly rather than ignoring it, and the explanation - a private well that provided a clean source of water - confirmed the theory precisely by explaining away the apparent counterexample.
|
The droplet/aerosol debate in 2020 had a Popperian structure too, but the falsifying tests were slow to arrive. The superspreader event in a Skagit Valley choir rehearsal in March 2020 (see right) - where 52 of 61 singers were infected after a two-and-a-half hour rehearsal, despite the absence of anyone visibly ill - was exactly the kind of unusual result that should have triggered urgent investigation of the droplet hypothesis.
It took months for the aerosol evidence to accumulate to the point where official guidance shifted. The problem was not that the falsifying evidence was hidden. It was that what TOK calls the 'community of knowers' were a very bad influence on each other. The scientific community was trapped in a questionable paradigm. You do remember the concept of paradigm shifts, right? |
|
Popper's insight looks even sharper in the age of artificial intelligence. Large language models - the systems behind ChatGPT, Claude, Gemini, and every other AI assistant you use - are trained on human approval. They learn by producing outputs that human evaluators rate as helpful, plausible, and satisfying. This means they are structurally optimised to confirm, to find reasons a claim sounds right, to produce answers that feel convincing, to agree rather than to challenge. This is precisely the habit Popper identified as epistemologically dangerous. Confirmations are easy and cheap. What distinguishes rigorous reasoning is the willingness to ask: what would show this to be wrong? An AI system trained to please human evaluators has been built to answer a different question entirely.
Big idea 4 - The social life of knowledge
Snow was right and the establishment was wrong. It took them a decade to accept it. Understanding why is as important as understanding Snow's method.
Snow was right and the establishment was wrong. It took them a decade to accept it. Understanding why is as important as understanding Snow's method.
"How could so many intelligent people be so grievously wrong for such an extended period of time? How could they ignore so much overwhelming evidence that contradicted their most basic theories? These questions, too, deserve their own discipline - the sociology of error."
Steven Johnson, The Ghost Map (2006)
Steven Johnson, The Ghost Map (2006)
The philosopher Thomas Kuhn, writing a century after Snow, gave the most systematic account of why Snow faced resistence for his theory. Scientific knowledge is not produced by individual researchers applying a universal method. It is produced by communities of knowers who share a paradigm - a set of assumptions, methods, exemplary problems and solutions, and a picture of what the world is like - that structures what they can observe, what questions they can ask, and what counts as an acceptable answer.
No scientist starts from first principles each time they enter a laboratory. Science requires shared assumptions about what counts as a valid method, a reliable instrument, and a meaningful result. Without them, inquiry cannot begin. Kuhn called this framework a paradigm: the background commitments that make normal science possible. The paradigm enables inquiry precisely because it is not itself under constant investigation. It tells you what to look for and what would count as an answer.
But what makes it productive also makes it resistant to challenge. When results do not fit the framework, they are not immediately treated as refutations. They are set aside, reinterpreted, or attributed to experimental error. A paradigm is not abandoned when evidence first goes against it. It is abandoned when the accumulation of anomalies becomes too great to contain, and a new framework emerges that can accommodate them better. Kuhn's deeper point is that this process is not purely logical. Paradigm shifts involve generational change in scientific communities and contests over institutional authority: who controls the journals and the funding, and who has the power to define what legitimate research looks like. You will remember that The Catholic Church did not reject Galileo's evidence because they examined it and found it wanting. They rejected it because it did not fit a framework they had no reason to question. That is what Kuhn means by normal science.
But what makes it productive also makes it resistant to challenge. When results do not fit the framework, they are not immediately treated as refutations. They are set aside, reinterpreted, or attributed to experimental error. A paradigm is not abandoned when evidence first goes against it. It is abandoned when the accumulation of anomalies becomes too great to contain, and a new framework emerges that can accommodate them better. Kuhn's deeper point is that this process is not purely logical. Paradigm shifts involve generational change in scientific communities and contests over institutional authority: who controls the journals and the funding, and who has the power to define what legitimate research looks like. You will remember that The Catholic Church did not reject Galileo's evidence because they examined it and found it wanting. They rejected it because it did not fit a framework they had no reason to question. That is what Kuhn means by normal science.
|
Peer review, replication, open publication, and structured debate are the social tools that scientific communities have developed to compensate for the fact that individual knowers are not neutral. These tools do not eliminate the problem of paradigm entrenchment, but they create the conditions under which revision becomes possible. Snow's vindication came not when he produced better evidence, but when germ theory provided an alternative paradigm that made his waterborne account coherent.
The early Covid consensus on droplet transmission was a paradigm in Kuhn's sense. It was not simply a claim about droplets. It was an entire infrastructure of public health guidance - hand-washing, surface disinfection, two-metre distancing - built on the droplet model. Abandoning the model required admitting not just that a theory was wrong, but that the guidance given to billions of people had been built on an incorrect framework. That is a much higher epistemological and political cost than simply updating a single claim. Kuhn understood that paradigms resist revision not because scientists are irrational, but because the rational and socio-political/cultural dimensions of knowledge production are inseparable.
|
Generated with Gemini - Click to enlarge
|
The mask guidance reversal was also, in part, a communication problem with an epistemological dimension. The early advice against masks was partly strategic and designed to protect the supply for healthcare workers. When the science changed, distinguishing what had been a justified scientific claim from what had been a strategic communication was nearly impossible from the outside. Also by this point there were enough masks to go around. This is what happens when the social and epistemic functions of knowledge-making institutions become entangled.
|
|
Paradigm Shift
Social Science Explainer Thomas Kuhn (1922–1996) was an American historian and philosopher of science best known for The Structure of Scientific Revolutions. He argued that science does not progress steadily but through disruptive “paradigm shifts,” where dominant frameworks are replaced after crises and anomalies accumulate. A paradigm shapes what counts as knowledge, methods, and truth; when it collapses, a new worldview emerges. Kuhn’s work challenged the idea of purely objective, cumulative scientific progress.
|
Bringing it together
The method and the tools are inseparable.
Go back to the clip you watched at the start. The Fauci interview was a claim made in good faith, on the basis of available evidence, within a paradigm that structured what counted as relevant and what did not. Hume tells us why more data alone was not sufficient: every inference from that data to a general claim about transmission was inductive, and induction cannot guarantee its conclusions. Popper tells us what should have happened faster: the anomalous superspreader events should have been treated as falsifying tests, not as noise. Kuhn tells us why they weren't: the droplet paradigm had become an infrastructure, embedded in labs, protocols, and funding decisions, and infrastructures resist revision.
Snow's story adds the tools dimension to the scientific method. His map did not just display information that already existed - it transformed it. The spatial visualisation made a pattern visible that the mortality tables could not show. This is a great TOK observation and also worth remembering in history when you're asked about the value of a graph. The absence Snow looked for at the workhouse was found because his hypothesis and map told him where to look. The pump handle was removed because he had a falsifiable prediction. This is what methods and tools in combination look like: a method (falsificationist inquiry) applied through specific instruments (the map, the anomaly-search, the controlled intervention) in a social context (the Board of Health, the competing paradigm, the eventual institutional acceptance).
Snow died in 1858, four years after the Broad Street outbreak, before germ theory was established and before his waterborne theory had been formally accepted. He had been right. He had done everything a careful empirical reasoner should do but he spent the last years of his life largely ignored. The process by which expertise generates and revises knowledge is messier, slower, and more socially conditioned than the clean image of scientific method suggests. Understanding that process is not a reason to distrust science. It is a reason to understand it more carefully.
The method and the tools are inseparable.
Go back to the clip you watched at the start. The Fauci interview was a claim made in good faith, on the basis of available evidence, within a paradigm that structured what counted as relevant and what did not. Hume tells us why more data alone was not sufficient: every inference from that data to a general claim about transmission was inductive, and induction cannot guarantee its conclusions. Popper tells us what should have happened faster: the anomalous superspreader events should have been treated as falsifying tests, not as noise. Kuhn tells us why they weren't: the droplet paradigm had become an infrastructure, embedded in labs, protocols, and funding decisions, and infrastructures resist revision.
Snow's story adds the tools dimension to the scientific method. His map did not just display information that already existed - it transformed it. The spatial visualisation made a pattern visible that the mortality tables could not show. This is a great TOK observation and also worth remembering in history when you're asked about the value of a graph. The absence Snow looked for at the workhouse was found because his hypothesis and map told him where to look. The pump handle was removed because he had a falsifiable prediction. This is what methods and tools in combination look like: a method (falsificationist inquiry) applied through specific instruments (the map, the anomaly-search, the controlled intervention) in a social context (the Board of Health, the competing paradigm, the eventual institutional acceptance).
Snow died in 1858, four years after the Broad Street outbreak, before germ theory was established and before his waterborne theory had been formally accepted. He had been right. He had done everything a careful empirical reasoner should do but he spent the last years of his life largely ignored. The process by which expertise generates and revises knowledge is messier, slower, and more socially conditioned than the clean image of scientific method suggests. Understanding that process is not a reason to distrust science. It is a reason to understand it more carefully.
A Spotter's Guide to Informal Fallacies
|
|
The Big Ideas in this lesson describe how good epistemological reasoning works - and where it breaks down. But bad arguments do not only fail because evidence is incomplete or paradigms are hard to shift. They also fail because they contain specific, identifiable structural errors - fallacies - that make them look more convincing than they are. Being able to name these errors is a practical skill. It is also essential preparation for the Social Sciences, where contested claims are everywhere and the quality of the reasoning behind them is always the first question to ask.
|
One observation worth making: several of these fallacies work together in practice. A single paragraph of motivated reasoning often contains a post hoc claim, a hasty generalisation, a circular structure, and an ad hominem attack simultaneously. The ability to identify them separately - to name precisely which step is fallacious and why - is what transforms vague unease about an argument into a specific, defensible critique. That is the difference between knowing something is wrong and being able to say why.
Fallacy |
What goes wrong |
Example |
Post hoc |
Assuming that because B followed A, A caused B. |
Cases rose after masks were introduced, therefore masks spread the virus. |
Hasty generalisation |
Drawing a broad conclusion from too few cases. |
Three vaccinated people caught Covid, therefore vaccines do not work. |
Straw man |
Misrepresenting an opponent's position to make it easier to attack. |
The scientists say masks are perfect protection, so when one study shows they are not, the whole case collapses. |
Circular argument |
Using the conclusion as one of the premises. |
The WHO guidance is reliable because the WHO is a reliable institution. |
Slippery slope |
Claiming that one step will inevitably lead to an extreme outcome without showing the causal chain. |
If we mandate one vaccine, governments will control all medical decisions forever. |
Ad populum |
Treating popularity or consensus as a substitute for evidence. |
Millions of people believe 5G caused Covid, so there must be something to it. |
Ad hominem |
Attacking the person making an argument rather than the argument itself. |
You can't trust Fauci's guidance - he has financial ties to pharmaceutical companies. |
Next: Lesson 6 - Methods and Tools 2 - How the Mind Actually Works
Lesson 5 showed how justification is supposed to work. Lesson 6 shows how it actually works - and why even careful, well-intentioned reasoners are systematically unreliable in ways they cannot detect from the inside.
Questions, assessments, films and other stuff.
Questions to think about
These are questions for discussion, reflection, and your TOK journal. They do not have single correct answers, they have better and worse arguments.
These are questions for discussion, reflection, and your TOK journal. They do not have single correct answers, they have better and worse arguments.
- The early mask guidance was wrong. Was it also unjustified - or was it a justified claim that turned out to be false? Is there a difference, and does it matter?
- Snow used a map; the miasmatists used mortality tables. Both were working with the same deaths. Did the tool change what the evidence could show - and does the choice of tool ever determine what a method can find?
- Snow's investigation succeeded because he looked for cases that should have appeared on the map but did not. Is looking for disconfirming evidence a general obligation for any knower, or is it specific to scientific reasoning?
- Hume argued that induction cannot be logically justified, yet we cannot do without it. Does this mean that all empirical knowledge is, at bottom, a matter of habit rather than reason?
- Popper argues that the more a theory forbids, the better it is. Does this criterion apply outside the natural sciences - in history, in ethics, in economics? If not, what replaces it?
- Kuhn and Popper disagreed about how science actually works. Popper believed scientists should try to falsify their theories; Kuhn observed that scientists rarely do this in practice. Which account is more epistemologically significant - the normative account of how science should work, or the descriptive account of how it does?
- The early Covid guidance on masks was partly strategic as well as scientific. When an institution gives guidance that is both a knowledge claim and a policy decision, how should we evaluate whether it was justified?
- Peer review, replication, and open publication are described as social tools of knowledge production. What happens to scientific knowledge when those tools malfunction - when peer review is captured by vested interests, replication is too expensive, or publication is paywalled?
Exhibition connections
See more exhibition ideas and previous student work here
It is never too early to start to think about your TOK Exhibition, the ideas in this lesson connect strongly to three of the 35 prompts. Start noticing objects in the world around you that speak to these questions.
Prompt #5: What counts as good evidence for a claim?
The lesson's central question. Snow's pump map was striking but insufficient on its own - the workhouse and brewery anomalies were what made the evidence compelling. The early Covid mask guidance was also based on available evidence; when that evidence was inadequate, the guidance failed. An exhibition object could be any piece of evidence whose meaning depends on the questions asked of it: a dataset, a map, a photograph, a scientific study. The connection lies in examining what makes it count as evidence for a particular claim - and what would count as evidence against it.
Prompt #19: What counts as a good justification for a claim?
Justification goes beyond evidence: it is the process of showing that the evidence supports the conclusion, by appropriate reasoning, to a sufficient degree. Popper's falsificationism gives one answer: a claim is well-justified if it has survived serious attempts to refute it. The mask guidance had not been tested against the superspreader evidence when it was issued. An exhibition object connected to a claim that has been tested and survived - a vaccine, a public health intervention, a historical account - allows exploration of what justification actually consists of.
Prompt #31: How can we judge when evidence is adequate?
Hume's problem means there is no formal answer to this question. In practice, adequacy is judged by communities of knowers applying shared standards - standards that are paradigm-dependent. The miasmatists and Snow were applying different standards for adequacy; both were doing so sincerely. The same was true of aerosol and droplet researchers in 2020. An exhibition object connected to a disputed question where evidence exists but is contested - climate science, a historical controversy, a public health debate - connects directly to the question of who decides when evidence is adequate, and on what grounds.
See more exhibition ideas and previous student work here
It is never too early to start to think about your TOK Exhibition, the ideas in this lesson connect strongly to three of the 35 prompts. Start noticing objects in the world around you that speak to these questions.
Prompt #5: What counts as good evidence for a claim?
The lesson's central question. Snow's pump map was striking but insufficient on its own - the workhouse and brewery anomalies were what made the evidence compelling. The early Covid mask guidance was also based on available evidence; when that evidence was inadequate, the guidance failed. An exhibition object could be any piece of evidence whose meaning depends on the questions asked of it: a dataset, a map, a photograph, a scientific study. The connection lies in examining what makes it count as evidence for a particular claim - and what would count as evidence against it.
Prompt #19: What counts as a good justification for a claim?
Justification goes beyond evidence: it is the process of showing that the evidence supports the conclusion, by appropriate reasoning, to a sufficient degree. Popper's falsificationism gives one answer: a claim is well-justified if it has survived serious attempts to refute it. The mask guidance had not been tested against the superspreader evidence when it was issued. An exhibition object connected to a claim that has been tested and survived - a vaccine, a public health intervention, a historical account - allows exploration of what justification actually consists of.
Prompt #31: How can we judge when evidence is adequate?
Hume's problem means there is no formal answer to this question. In practice, adequacy is judged by communities of knowers applying shared standards - standards that are paradigm-dependent. The miasmatists and Snow were applying different standards for adequacy; both were doing so sincerely. The same was true of aerosol and droplet researchers in 2020. An exhibition object connected to a disputed question where evidence exists but is contested - climate science, a historical controversy, a public health debate - connects directly to the question of who decides when evidence is adequate, and on what grounds.
Feature (and documentary) films
For more see my 10 films for the TOK journey page.
For more see my 10 films for the TOK journey page.
🎬 WATCH - Contagion dir. Steven Soderbergh (2011)
An unusually accurate dramatisation of epidemiological investigation. The scientists tracing the MEV-1 transmission chain are doing exactly what Snow did - mapping cases, looking for anomalies, testing hypotheses against disconfirming evidence. Made before Covid-19 and eerily prescient about it: the film's depictions of institutional resistance, public misinformation, and the gap between what scientists know and what governments communicate became a reference point during the pandemic. The methodology scenes - particularly those involving Laurence Fishburne and Kate Winslet - directly illustrate the lesson's four big ideas. My students can watch the film here.
An unusually accurate dramatisation of epidemiological investigation. The scientists tracing the MEV-1 transmission chain are doing exactly what Snow did - mapping cases, looking for anomalies, testing hypotheses against disconfirming evidence. Made before Covid-19 and eerily prescient about it: the film's depictions of institutional resistance, public misinformation, and the gap between what scientists know and what governments communicate became a reference point during the pandemic. The methodology scenes - particularly those involving Laurence Fishburne and Kate Winslet - directly illustrate the lesson's four big ideas. My students can watch the film here.
|
|
🎬 WATCH - Totally Under Control dir. Alex Gibney (2020)
A documentary examining the US government's response to Covid-19 in the first year of the pandemic, with particular attention to the gap between available evidence and official guidance. Gibney's method - interviewing scientists, public health officials, and policymakers - makes the epistemological questions visible: when did the evidence change? When did the guidance change? Why was there a lag? These are Kuhn's questions about paradigm resistance made concrete and recent. (Available on Amazon and Apple)
A documentary examining the US government's response to Covid-19 in the first year of the pandemic, with particular attention to the gap between available evidence and official guidance. Gibney's method - interviewing scientists, public health officials, and policymakers - makes the epistemological questions visible: when did the evidence change? When did the guidance change? Why was there a lag? These are Kuhn's questions about paradigm resistance made concrete and recent. (Available on Amazon and Apple)
|
|
Further reading
Books
📚 READ - The Ghost Map by Steven Johnson (2006)
The spine of Big Idea 1 and the guide for the whole lesson. Chapters 1 to 5 cover the investigation itself - Snow's background, the outbreak, the mapping, the disconfirming cases, the Board of Health. The book reads at pace and can be completed in a weekend. Johnson's method - using a single episode to illuminate large questions about how knowledge is produced - is itself a model of how history and epistemology can be combined.
📚 READ - Matthew Syed, Black Box Thinking (2015)
An argument for why some institutions learn from failure and others do not. Syed's central comparison is between aviation, which treats every crash as data to be understood and shared, and medicine, which has historically treated failure as a professional embarrassment to be minimised. The book is Popper's falsificationism applied to institutional design: the question is not whether a system produces errors, but whether it is structured to surface them. Chapter 2, on the black box itself, and chapter 5, on the culture of denial in medicine, are the most directly relevant to this lesson.
📚 READ - Conjectures and Refutations by Karl Popper (1963)
The Introduction and Chapter 1 ("Science: Conjectures and Refutations") are accessible without a scientific or philosophical background and contain Popper's clearest statement of the falsifiability argument and its relationship to the problem of induction. The three numbered theses - confirmations are easy; confirmations only count if they result from risky predictions; a good theory forbids things - can be read, understood, and applied in an afternoon.
📚 READ - The Structure of Scientific Revolutions by Thomas Kuhn (1962)
Chapters 1 to 4 introduce normal science and the paradigm concept; Chapter 6 on anomaly and discovery is the most directly relevant to this lesson. Kuhn writes clearly and his examples - from Copernicus to Lavoisier to the discovery of X-rays - are chosen for their illustrative power. Chapter 6 can be read standalone.
The spine of Big Idea 1 and the guide for the whole lesson. Chapters 1 to 5 cover the investigation itself - Snow's background, the outbreak, the mapping, the disconfirming cases, the Board of Health. The book reads at pace and can be completed in a weekend. Johnson's method - using a single episode to illuminate large questions about how knowledge is produced - is itself a model of how history and epistemology can be combined.
📚 READ - Matthew Syed, Black Box Thinking (2015)
An argument for why some institutions learn from failure and others do not. Syed's central comparison is between aviation, which treats every crash as data to be understood and shared, and medicine, which has historically treated failure as a professional embarrassment to be minimised. The book is Popper's falsificationism applied to institutional design: the question is not whether a system produces errors, but whether it is structured to surface them. Chapter 2, on the black box itself, and chapter 5, on the culture of denial in medicine, are the most directly relevant to this lesson.
📚 READ - Conjectures and Refutations by Karl Popper (1963)
The Introduction and Chapter 1 ("Science: Conjectures and Refutations") are accessible without a scientific or philosophical background and contain Popper's clearest statement of the falsifiability argument and its relationship to the problem of induction. The three numbered theses - confirmations are easy; confirmations only count if they result from risky predictions; a good theory forbids things - can be read, understood, and applied in an afternoon.
📚 READ - The Structure of Scientific Revolutions by Thomas Kuhn (1962)
Chapters 1 to 4 introduce normal science and the paradigm concept; Chapter 6 on anomaly and discovery is the most directly relevant to this lesson. Kuhn writes clearly and his examples - from Copernicus to Lavoisier to the discovery of X-rays - are chosen for their illustrative power. Chapter 6 can be read standalone.