Original episode & show notes | Raw transcript
The podcast opens by criticizing the definitive statement, “the science says.” The hosts, all with backgrounds in science and coaching, argue that this phrase is an oversimplification and often misrepresents the nature of scientific inquiry. Here’s why:
Science is a Process, Not a Monolith: Science is an ongoing process of investigation, not a static body of facts. A single study is a snapshot, not the final word.
Overstated Certainty: The phrase implies a level of certainty that rarely exists, especially in biological sciences where individual variability is high. The hosts, despite their expertise, express a lack of confidence in making such definitive statements.
Lack of Context: Scientific findings are highly contextual. A result from one study might only be true under very specific conditions (e.g., for a particular population, at a certain time of year, with a specific training history).
The hosts highlight several inherent limitations in sports science research that consumers of this information should be aware of:
Small Sample Sizes: Many studies use a small number of participants (e.g., 8-12 people), which limits the statistical power and generalizability of the findings.
Participant Characteristics: The subjects of a study are often a very specific group (e.g., untrained individuals, moderately trained, or elite athletes from a national federation). Results from one group cannot be directly applied to another. For instance, a training intervention that works for detrained individuals early in their season will likely have a different effect on a well-trained athlete mid-season.
Study Design and “Dirty Laundry”: Scientists are aware of these limitations. The “dirty laundry” of science is that the controlled environment of a study often doesn’t reflect the complex, messy reality of an athlete’s life, which includes factors like stress, diet, and sleep.
Temptation to Extrapolate: There is a strong temptation for both researchers and the media to extrapolate a narrow result and apply it broadly, which is often not supported by the data.
A significant portion of the discussion is dedicated to the importance of understanding statistics to properly interpret scientific papers.
Beyond Averages: Papers typically report average responses, but the variance or standard deviation is crucial. This tells you about the spread of results and proves that individual variation exists. Some individuals might be “hyper-responders,” while others might not respond at all or even respond negatively.
Statistical Significance vs. Practical Importance: A result can be statistically significant (unlikely to be due to random chance) but not practically meaningful. For example, a 2-watt increase in FTP might be statistically insignificant due to the error range of power meters.
The “Five Sigma” Standard: In fields like particle physics, a “five sigma” result (about a 1 in 3.5 million chance of being random) is the gold standard for a discovery. In biology and exercise science, the bar is much lower (often a p-value of less than 0.05, or a 1 in 20 chance of being random). A “three-sigma” result in exercise science would be considered a home run.
Misleading Visuals: Be wary of how data is presented. Truncated axes on graphs can make a small, insignificant difference appear massive.
The central theme that emerges is the necessity of individualizing training.
The Fallacy of Division: This logical fallacy occurs when one assumes that what is true for the whole is true for every part. Just because a study’s average result is positive does not mean the intervention will work for every individual. The podcast uses the analogy of a class on average liking cake, while “Timmy is a pie guy.”
N=1 Experimentation: Every athlete is an “N of 1” experiment. The role of a coach is to use scientific literature as a starting point to form hypotheses and then test them with the individual athlete.
Evidence-Based Practice: True evidence-based coaching is not about blindly following a study protocol. It’s about collecting evidence from the athlete. If an intervention works, that’s evidence. If it doesn’t, that’s also evidence. It’s a continuous process of “fuck around and find out,” albeit from an informed starting position.
The Ramp Test Example: The podcast uses the ramp test as a prime example of where population averages fail. The common formula (e.g., 75% of the final minute’s power) is an average. However, an individual’s actual threshold can vary from 55% to 90% of their ramp test result, depending on their unique physiology, particularly their anaerobic capacity. A single data point that falls outside the “accepted range” invalidates the universal applicability of the test.
The hosts also shed light on the ecosystem of scientific publishing and media, which influences how science is communicated.
Publication Bias: Journals are more likely to publish positive, “flashy” results than null results (where an intervention showed no effect). This can skew the public’s perception of the literature.
Science Communication: There is often a gap between the nuanced understanding of the researchers and the simplified message that reaches the public. This is sometimes the fault of science communicators or media outlets who may lack a deep scientific background or who prioritize clicks over accuracy.
Clickbait and Financial Incentives: The “capitalist model of social media” incentivizes sensationalism. Clickbait titles and splashy thumbnails are designed to grab attention, and the content may not always live up to the hype or provide the necessary context and duty of care (e.g., explaining the risks of a technique like Blood Flow Restriction).
Paywalls and Access: Many scientific papers are behind expensive paywalls. While the hosts humorously refuse to condone illegal means of access, they note that authors are usually happy to share their papers if you email them.
The podcast concludes with a discussion on where the field should go.
Greater Transparency: A desire for more honesty about what a paper is and isn’t saying. This includes publishing individual participant data (even in supplementary files) so that others can see the full spectrum of responses.
Better Study Designs: A need for studies designed to investigate individual responses. The “holy grail” would be to identify characteristics that predict who will respond best to which type of training.
Improved Science Communication: A call for more scientists to engage with the public to provide context and for a more critical and educated consumer base.
Reproducibility: Acknowledging the “reproducibility crisis” in science, where it’s estimated that a large percentage of studies cannot be replicated by other labs. This is a major challenge that the field needs to address through more detailed and transparent methods.
In essence, the podcast advocates for a healthy skepticism and a deep appreciation for nuance. It encourages listeners to view science not as a rulebook, but as a toolbox and a starting point for their own informed, individualized journey in sport.